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This working note contains descriptions of the methods that the 
Goddard Space Flight Center of NASA, the National Cancer Institute of 
the National Institutes of Health, and the U.S. Air Force use in man- 
aging programmatic research and development. These descriptions were 
obtained by interviewing managers and other staff personnel in these 
agencies and interpreting their responses. None of the descriptions 
have yet been returned to the agencies for their comment or approval. 
This will be done in the near future. In addition, the section con- 
tains an overview of quantitative models used by some industrial 
companies and governement agencies to evaluate and choose among pro- 
ject proposals. 

This work was done as part of the effort to plan the National 
Institute of Education (NIE) . If authorized by the Congress, the NIE 
would conduct research and development in the field of education. This 
report is one of a series on the Institute. The others are: 

^ National Institute of Education: Preliminary Plan for the 
Proposed institute (R-657-HEW) 

National Institute of Education: Methods for Managing Funda- 
mental Research (WN-7676) 

National Institute of Education: Methods for Managing Practice 

oriented Research and Development (WN-7677) ' 
° National Institute of Education: Organizational and Managerial 

Alternatives (WN-7679) 
^ National Institute of Education: Evaluation of Methods for 

Maraging Research and Development (WN-7680) 

Tl-iis report describes methods that the selected agencies use 
in managing only programmatic R&D; it does not evaluate the relative 
merits of these methods. A comparative evaluation appears in WN-7680. 

Many of the R&D managers interviewed during this study expressed 
the need for additional study of the methods used in managing non- 
military R&D in the federal government. This series of reports seeks 
to provide a basis for research into improved management practices 
for that entire area. The principal purpose of these reports, however. 



is to enable the planners of the National Institute of Education to 
benefit from the experience of other federal R&D agencies in develop- 
ing the NIE^s R&D management procedures. 
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I , INTRODUCriOK 

PURPOSE OF THIS REPORT 

This report describes four different management systems that federal 
agencies use in managing 'progTcomatip researdh and developrnent. The 
first three Indicate the procedures/used by their respective agency; 
the last outlines the principlesy^f quantitative models applicable for 
part of the management process. 

For the first three, the format of description will be to treat 
one agency at a time by (1) presenting the steps that its managers and 
researchers actually take in managing research activity, and then (2) 
repeating this presentation for the other agencies. The intention is to 
present the data on which the interpretive and evaluative statements 
made in WN-*7679 and Wn-7680 are based. This approach was taken because 
agreement on what people do is easier to get than agreement on the 
effects of their actions, or what they should do. Agreement on the 
effects of their actions is lacking because insufficient research on 
the management of federal R&D has been done. Agreement on what R&D 
managers should do is difficult because it is ultimately a question of 
value. 

The description for each agency and for the quantitative models 
will be called a paradigm^ since not every detail and variant in what 
is actually done will be described. Each description is meant only 
as a model that depicts the essential steps in the R&D management 
process. 

A step is deemed essential if changing it would significantly alter 
an estimate of the basic philosophy underlying the R&D management pro- 
cess being described. By looking at the essential steps , it is easier 
to infer what the basic underlying philosophy is, and how to project it 
onto a new situation, such as education R&D. This is, in fact, a 
meaningful definition of a paradigm; that is, the projection of some- 
thing which is difficult to describe dnto reality, where its consequences 
are observable and hence describable. 

The paradigms that will be treated in this report are listed in 
Table 1. 

O 
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Table 1 

PARADIGMS DESCRIBED FOR MANAGING 
PROGRAMMATIC RESEARCH iiND DEVELOPMENT 

The Management Paradigm used by the Goddard Space Flight Center of the 
National Aeronautical and Space Administration 

The Management Paradigms used by the National Cancer Institute for 
conducting National Research Programs 

The Management Paradigm of the U.S. Air Force for conducting 
Major Weapon System Development 

The Paradigm of Quantitative Models for Project Selection used in 
Industry and Governi^\ent Agencies. 



METHOD OF RESEARCH 

The data used to construct the management paradigms were obtained 
by interviewing federal R&D managers and reviewing agency documents and 
literature* For each agency > key personnel and those recommended by 
key personnel were approached for interview. Altogether 14 managers 
were interviewed for this one report, some on repeated occasions. A 
list of the people interviewed will appear in the final /ersion of this 
report . 

The paradigms are a distillation of replies made by managers comm- 
enting on the nature and relative importance of their various activities. 
Necessarily, this approach to research is vulnerable to biases and some- 
times produces information that is difficult to verify. Nevertheless, 
by asking all managers similar questions, and by filtering the responses 
as objectively as experience made possible, a fair representation of 
reality 11^ thought to be presented. This approach is within the tradi- 
tion of naturalistic observation as a method of research. 

To gain clarity of exposition, some of the auxiliary mechanisms 
used by some agencies to overcome shortcomings in their management pro- 
cesses were omitted. Thus, matching the paradigm descriptions, the 
agencies interviewed, and the paradigm evaluations In tfN-7680 to conclude 
that one agency does a better job of managing research than another 
agency Is not justified. The operations and usefulness of these auxil- 
iary mechanisms are discussed in the evaluation report, WN~7680. 
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TYPES OF R&D ACTIVITY 

This report Is limited to the management of programmatic research 
and development. Methods for managing two other kinds of R&D activity, 
fundamental research and practice-oriented R&D, appear in UN- 7676 and 
WN-7677 respectively. 

In simplest terms, programmatic R&D is activity undertaken to 
accomplish a specific objective relating to a particular problem. It 
is usually development oriented, and its success is measured by the 
accomplishment of a capability or task rather than solely by standards 
of disciplined inquiry using judgment by scientific peer groups. 

Programmatic R&D is goal-oriented rather than conclusion-oriented. 
It results not in statements of truth but in actions or products or 
decisions for action. 

Another distinguishing characteristic of programmatic R&D is 
that the component project activities are not ones chosen and directed 
independently by the researchers themselves, but are activities which 
fit within a predefined overall program plan, whether formally con- 
structed or informally visualized by program directors. This program 
plan serves to relate component projects and guide the selection of 
future work by providing criteria of relevance, logical priority, and 
program need. For these reasons, programmatic R&D is often funded by 
contracts rather than grants, stressing control, relation to predefined 
problem, ant! attention toward cost/benefit and time considerations as 
well as tec-mical and scientific excellence as judged by other scientists. 
This does not me^ui, however, that ideas of potential high scientific 
merit yet low relevance to a selected approach are automatically dis- 
carded. New approaches to a problem, new technical breakthroughs In 
surrounding areas of inquiry > and new problems themselves > often re- 
directed efforts of programmatic R&D resulting in new formulations of 
program plans and new criteria for relevance and benefit. 

TYPES OF MANACTIMENT ACTIVITY 

To facilitate presentation, the research management process will 
be broken into three types of activity: 

O 
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® Program Planning, 
^ Program Development, 
® Program Evaluation. 

These categories are deliberately chosen to group together qualitatively 
similar management activities. 

Program Planning management activity is defined to include all the 
actions tcV.en to foster, detect, and formulate new programs of research 
and development. Also included are the procedures for deciding which 
new subprograms will be added to the set of ongoing streams of activity. 
One example of such a new program is NCI's effort to mount a coherent 
attack on breast cancer > from detection to treatment, and including 
vlrol, chemical, and immunological approaches. The plan for such a 
program does not merely indicate what research needs to be done, 
but relates projects together in a logical and time sequenced maimer, and 
identifies the managerial decisions which must be made between all 
major phases of activity. These characteristics are typical of plans 
associated with most efforts of programmatic R&D. 

Program Development is defined to be the activity of managing the 
continuous process of refining and elaborating the plan of programmed 
activity. As a management process. Program Development is typically, 
though not always, an iterative and continuing seqiwnoe of stages 
involving: 

^ assessment of program progress and needs, 

^ generation of project ideas , 

• selection of projects to support, 

® monitoring of project performance, 

^ evaluation of project outcomes , and 

^ utilization of project results. 

In various management paradigms, these stages are managed in 
different ways. Sometimes they are done by Program Directors, sometimes 
by project-specific temporary organizations or boards and sometimes by 
the overall Agency Director. In most of R&D, and especially when pro- 
grammatic R&D is being done, action proceeds concurrently in several 
of the project stages for different projects within the program. This 
activity is called Program Development because from program management's 
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perspectlve, progranmatic efforts require the combiiied and integrated 
results of component projects, selected initially and during the course 
of a program, to achieve its overall objective. 

. Program Evaluation is the management activity of assessing what 
has been accomplished at some point in time by the summation of results 
from all component projects. The way in which agencies accomplish 
this management activity is the third topic to be discussed in this paper. 

Discussion of management methods will focus on what is done at 
the program director's level, and only occasionally at higher levels. 
This is necessary because of the decision to describe procedures that 
managers use in practice. Interactions at the higher levels are more 
political, and thus subject to greater variation and personality depen- 
dencies. Not much insight into ways of allocating a budget between 
programs of one sort or another is gained by looking at the procedures 
agencies use. The influences surrounding the making of these kinds of 
decisions is the subject of studies In the policy and political science 
literature. Attention here will center on what occurs at the interface 
between the performer and his more Immediate managers. 
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IT -PROGRAMMATIC PARADIOl OF NASA/GODDARD SPACE FLIGHT CENTER 
OVERVIEW 

Goddard's strong linkage between intramural and extramural acti- 
vities is the result of a major redirection toward programmatic, 
development projects for what was once a basic research facility. 
NASA was originally an outgrowth of NACA (National Advisory Committee 
for Aeronautics) laboratories. NACA sponsored primarily basic research 
which was litiited to the field of aeronautics. In response to the 
Sputnik challenge of 1957, NASA was created under the Eisenhower admin- 
istration to build the nation's competency in aerospace as well as 
aeronautics. 

With the expansion of objectives into aerospace, grants and con- 
tracts were given by NASA to universities and industries to supplement 
the existing facilities of NACA I.abs. The old NACA Labs remained 
relatively intact , but a new extramural branch was established in NASA 
to manage the extramural work. Both parts operated independently of 
each other. The arrangement was short-lived. 

Goddard came into being in 1959 as the first NASA center devoted 
strictly to space flight. Personnel were recruited from three sources: 

(1) the Signal Corp, concerned primarily with meteorlogical programs, 

(2) the NRL (Naval Research Laboratory), concerned with launching a 
satellite as part of a world wide effort at the time, and (3) from NACA 
Labs at Lewis and Ames Research Centers. NASA's original emphasis on 
basic research, inherited from the former NACA Labs, along with a 
separate NASA branch for extramural contract management, seemed no long- 
er appropriate for Goddard's well-defined and programmatic objectives 
for research and development. Though Congress was financially uncon- 
straining at the time. It looked for quick results. To meet its new 
objectives, Goddard had to meld together its intramural and extramural 
program activities. 

To provide an organizational linkage, Goddard created a matrix 
form of program management which Interweaves intramural researchers from 
discipline-oriented technology units with project-focused extramural 

O 

ERIC 



-7- 

management teams. This form of management has two essential require- 
ments for success: (1) a strong intramural competency for program 
management, and (2) a diversity of both program interests and capa- 
bilities. Both are needed to maintain skills in managing and eval** 
uating extramural programs. These skills are paramount for Goddard's 
success. Unlike the situation in fundamental research, in which outside 
scientists are called upon to help make decisions about grants to 
their fellow scientists, it would be inappropriate for Goddard to rely 
upon engineers from industry to help in making contract decisions. 
Consequently, Goddard must have the inhouse competency to make these 
contract decisions itself. At the program ^el, however, beyond the 
immediate realm of contract decisions, outside reviewers are employed 
to assist in program evaluation* 

Goddard is primarily a development organization. With four thou- 
sand employees, it currently manages thirty-eight major projects with 
an average total product cost of about fifty million dollars. Dom- 
inating its program development activity is a management procedure 
emphasizing sequential phases of program planning, committment, and 
management. Most new projiects fit under the broad objectives of 
relatively stable, large efforts such as the Physics and Astronomy 
program, Lunar and Planetary program. Communications program, and 
the Earth Observations program, though some projects may lead to new 
major programs as well. Small projects are managed by about twenty 
people, large ones by between forty and fifty people 

Flight project R&D constitutes about sixty percent of Goddard's 
program budget, and most of new project development. The other forty 
percent is used largely for programs in Tracking and Data Acquisition, 
which entail operation of world-wide networks for communication and 
satellite tracking. Since the main focus of this report is on program 
planning and development rather than on operations, only the Goddard 
management procedures for conducting flight project R&D will be dis- 
cussed. Essential features of this management paradigm are: 

o Intramural and extramural activities are integrated^ primarily 
through a matrix form of project staffing. 
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o Project proposals are reviewed internally by formal 

Source Evalmtion Boarda. 
o Decisionmaking for project selection follows a sequential 

process of atudy and project phases ^ each successive phase 

representing a higher committment by Goddard to provide 

full project funding, 
o Training and recruitment policies emphasize maintaining a 

strong inhouae competence for both research and management, 
o Contracts for projects are awarded mostly to industrial 

organisations, though sometimes awards may be inade to 

individual scientists who then subcontract out to other 

agencies • 

o Most managerial man-hours are devoted to managing existing 
contracts, though a considerable amount of time and effort 
is directed to project planning. 



SUMMARY 

General Characteristics 

Primary output of 
flight project R&D: 

Mechanisms of support: 



Staffing plan: 



ERIC 



Development of unmanned satellites 
for research and application. 

Planning and development contracts. 
Though Goddard contracts are generally 
performed extramurally , the prepara- 
tion of technical and management 
plans y the preliminary research on 
alternative approaches, the specifica- 
tion of system design, and the actual 
system development are all managed 
intramuralty . 



Matrix form, in which project manage- 
ment offices frequently cut across 
discipline-oriented directorates , 
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Program Planning 

Source of new 
project Ideas: 



Mechanisms for 
planning: 



Coordination: 



though some projects may exist entirely 
within a single directorate. All managers 
at the systems level of a project are 
transferred to the projects office, 
full-time and co-located. At the 
subsystems level for small projects, 
and at the components level, managers 
stay within their own directorate 
while working on a project. An admin- 
istrative officer is assigned full-time 
to each project, bringing with him a 
team of administrative service personnel. 



Major programs are relatively broad and 
stable. Ideas for new programs may 
stem from project plans or project con- 
tracts, managed inhouse. Most new programs 
originate from ideas of individual intra- 
mural researchers, though some may be 
initiated at the request of NASA Head- 
quarters, or advisory boards consisting 
mainly of outside scientists. 

Each year, the NASA Program Administrators 
review the objectives and overall R&D 
thrust of the major program areas, and 
communicate any revisions desired to the 
appropriate NASA centers. Goddard*s 
short-range (one year) program plans 
are subsequently prepared by inhoutse 
study groups and approved by the Goddard 
Management Council. 

Program areas are coordinated by the 
Goddard Management Council which meets 
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Program Developmen t 

Sources of project 
Ideas : 



Evaluation of 
project plans: 



Determination of 
contract type and 
competition: 



together one day every week. Council 
members are the heads of Goddard direct- 
orates. 



Inhouse researchers outline objectives 
for new project possibllitio.s on ideas 
lisually of their own selection. These 
outlines are essentially proposals of 
studies for potential projects. 

In the recent past, project ideas went 
through four major phases of develop- 
ment and review: Phase A activity (study) 
requiring approval by the Goddard Center 
Director; Phase B activity (alterraative 
approaches and system design) requiring 
approval by the Center Director; Phase C. 
activity (development of the Project Plan) 
requiring approval by both the Center 
Director and Management Council; and 
finally Phase D activity (actual system 
development) requiring review and approval 
of the Project Plan by the NASA Direc- 
tor prior to any RFP. Currently, Goddard 
management is considering collapsing these 
four phases into three. 

The type of contract and degree of 
contract competition depends primarily 
on the relevant phase of project activity. 
Contracts for Phases A and B are fixed 
price or cost plus fixed fee,, and for 
Phases C and D are incentive types. 
Contracts may be awarded to single or 
multiple contractors for alternative 
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Evaluatlon of 
project proposals: 



Monitoring of project 
performance: 



Evaluation of 
projects; 



approaches, and all Involve open 
competition except for few cases of 
Phase A work where awards may be made 
by the Goddard contracting officer on 
a noncompetitive basis. 

For all competitive proposals, review 
and evaluation is made by a formal 
Source Evaluation Board constituted 
separately for each project over one 
million dollars. Board members include 
Goddard personnel, representatives from 
NASA Headquarters , and occasionally out- 
side government consultants. Proposal 
ratings are made by Technical and Bus- 
iness Management committees operated 
independently of each other using 
criteria predefined by the Source Evalr 
uation Board. Board evaluations are 
presented to a formally appointed 
"Selecting Official" for final contract 
decisions. Depending on project cost, 
this official may be either the NASA 
Administrator or the Goddard Director. 

Extramural projects are managed by the 
Goddard Projects Directorate. Intramural 
projects are managed by the Goddard 
Technology Directorate. 

Project evaluations are conducted as an 
ongoing part of project activities rather 
than at the completion of the project. 
Monthly reviews by project managers com- 
pare technical performance, cost, and 
manpower required with benchmark figures 
indicated on the Project Plan. 
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Program Evaluation 



Mechanisms of 
evaluation: 



Progra© level reviews are conducted 
by Goddard Management Council and the 
NASA Program Administrators. Here again, 
program evaluations are actually summar- 
ies of ongoing project evaluations rather 
than special effort.? conducted at program 
conclusion. 



ACTIVITIES 
Program Planning 

Annual review of NASA's overall objectives and prograK thrust pro- 
vides the central guidelines for Goddard program plannin,^. This NASA re- 
view encompasses the programs of all NASA centers. Goddard has made 
some attempts at long-range planning in the past, but time and resource 
pressures have kept them to short-range yearly planniixg. 

The yearly revised program plans for Goddard are prepared by the 
Applications or Science Study Groups consisting of key scientists 
within the center. These groups propose to the Goddard Management 
Council what the short range objectives of Goddard should be. Approved 
program plans are then distributed by the Council to center managers. 

Project Initiation 

Major programs at Goddard generally develop from individual pro- 
jects which reveal new areas of interest or new means of research in 
aerospace. At any tine, an individual or team of intramural researchers 
may initiate a phased process of project development. In the recent 
past, this process has consisted of four distinct phases: 
Phase A: Preliminary Analysis (The Idea Stage) 
Phase B: Definition (Alternative Approaches) 
Phase C: Design (System Specification and Management Plan) 
Phase D: Development/Operations (Hardware Fabrication and Use) 
The sequential nature of phased project activity highlights the Goddard 
philosophy of step-wise increasing committment to an idea. At the end 
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o£ each phase, the project idea Is evaliiated and approved for further 
development by the Center Director, and also by the Management Council 
for Phases C and D. Between phases, potential projects are evaluated on 
the following criteria: (1) Timing: Is the time ripe to obtain Con- 
gressional support for the project missicnV (2) Manpower requirements: 
Does Goddard possess the needed resources to initiate and manage the 
required effort? and (3) Attractiveness. Does the potential project 
complement the Goddard portfolio? Will it maintain diversity of 
interest and capitalize on Goddard *s assets? 

Ultimately, the NASA Administrator authorizes all Goddard projects, 
but the phased project activity process is specific to Goddard manage- 
ment. Historically, passage through all four phases has taken an 
average of approximately seven years, with a great amount of time con- 
sumed by the decision processes between phases. Phases A and B are 
under the jurisdiction of Goddard *s Study Management System. With 
the approval for Phase C activity, the study takes on a project status. 
Recently, there is a tendency to combine Phases C and D for new projects. 
Detaj.ls of each phase are presented belc^. 

Pre-Phase A activity ^ consisting of ad hoc, unsolicited formula- 
tions of potential projects, is part of Goddard *s continual research 
efforts. This activity is directed toward identification of new 
objectives or missions, and culminates in Phase A Project Plans » 
typically from 3-8 pages, submitted to Goddard 's Management Council. 
If the plan calls for less than two man-years of effort in Phase A 
activity, it may be approved by the division head of the initiating 
researcher. Otherwise, approval from the Center Director is required. 

Approval to conduct Phase A activity is followed by the appoint- 
ment of a Study Manager, commonly the orginator of the project idea. 
Phase A activity averages three to six months duration, is generally 
conducted intramurally , and includes development of a concise state- 
ment of mission objectives , identification of research and technical 
requirements, and assessments of feasibility and desiraVility of further 
definition. Sketches of alternative approaches and candidate experi- 
ments for fulfilling mission objectives are included. No specific 

funds at Goddard are allocated toward Phase A work. 
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Authorization by the Center Director for Phase B activity occurs 
for only about ten projects per year. Because the planned activity for 
Phase B is relatively large^ usually from six to twelve months time and 
requiring ten to fifty men, Congress and O.M.B. are often interested. 
Pliase B Project Plans, typically from 20-30 pages, are forwarded to 
NASA Headquarters and at this time the study is elevated to "new start" 
project status. Phase B activity is mostly concerned with the devel- 
opment of alternative ways of achieving the mission objectives and the 
identification of the state-of-the-art constraints. Most activity 
is conducted intramurally, though for some projects, about half the 
activity is contracted out. Estimates of development time and total 
runout costs calculated during this phase become the base figures for 
any subsequent evaluations. Final Phase B activity is the produc- 
tion of a Phase B Analytical Report and a Phase C Project Plan. 

All Phase C Project Plans are reviewed and critiqued by the head of 
Goddard's Projects Directorate. Approval for Phase C activity by the 
Center Director and Goddard Management Council elevates the project 
idea from"study" to "project" status. This phase includes detailed 
system design, mockups» and identification of backup systems and their 
development requirements. Generally » all Phase C or Phase D work is 
contracted out. In specific cases where Goddard managers feel they 
must develop or maintain a certain competence within an area of inter- 
est, Goddard may conduct the entire project totally Intramurally . 

Typically thirty to forty-eight months are required from the end 
of Phase B to project completion. Phase C activity averages two to 
three years. Phase C final reports, submitted to NASA Headquarters, 
are directed primarily toward recommendations for subsequent actions. 
An overall description of the project planning process at Goddard 
appears in Figure 1 . 

Contractor Selection 

Procedures for the evaluation of external proposals for Goddard 
contracts vary according to the different activity phases. Descrip- 
tions for each phase of (1) the number of contracts written, (2) the 
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nature of contract competition, (3) the type of contract, (4) the 
use of RFPs, and (5) the responsibility for contractor selection are 
Illustrated in Figure 2« Except for some instances of Phase A work 
which is awarded on a noncompetitive basis by a Goddard contract* 
Ing officer, all contracts are competitive based, involve KFPs, and are 
reviewed for evaluation by a formal Source Evaluation Board consti- 
tuted separately for each projecu over one million dollars. 

Members of the Source Evaluation Board include Goddard personnel, 
representatives of NASA Headquarters, and occasionally government 
consultants. No representatives of potential industrial contractors 
are included. Prior to the distribution of lUFPs, the Board prepares 
a list of the criteria by which all proposals will be evaluated. When 
proposals in response to the KFPs cone in, they are given to two sep- 
arate committees of the«Board, the Techz^.ical Committee and the Business 
Management Committee, for independent assessments In those respective 
areas. The two committees use a complex rating system and operate with 
separated proposal data so that neither committee knows how any single 
proposal rates in the area of the other committee. Findings of the 
committees are presented to the Source Evaluation Board which In turn, 
submits its own evaluations to a "Selecting Official" who has final 
authority for selection decisions. Generally, if the cost of the 
project is over five million dollars « the NASA Administrator serves as 
Selecting Official. For projects of less cost, authorization for 
selection rests with the Goddard Director. 

Depending on practical and funding limitations , contracts for 
Phases A through C may be awarded to multiple contractors for alter- 
native approaches , and may involve either equal or unequal amounts of 
funding. Unequal amounts may be awarded because of differing competitive 
cost positions, or different amounts of work needed to be performed 
by the different contractors. 

For Phase B work, Goddard may allow contractors to submit pi^oposals 
on either: (a) one or more approaches to be studied, or (b) all 
approaches to be studied. If there appears to be a lack of con- 
tractors capable of performing studies on all approaches, 
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Goddard contracting officers will probably choose the first alternative. 
On the other hand» If Goddard officers expect an unwarranted . .luctance 
on the part of contractors to propose on a specific approach » they 
may choose the second alternative. For both Phase B and A work» 
contracts are either fixed price or cost plus fixed price. 

Generally two, and sometimes more than two, prime contractors are 
selected for Phase C work. Only those contractors capable of performing 
through Phase D are considered eligible for Phase C contract awards. In 
most instances, one of the prime contractors engaged in the Phase C 
activity will be selected for the Phase D award. Varioxis types of 
Incentive contracts are used for Phase C and D work, thougih some cost 
plus fixed fee contracts have been used. Preferably, the same Source 
Evaluation Board personnel will be used for Phase C and D evaluations. 
Oftentimes, the Goddard project management personnel are used as consultants 
to the Board during the evaluation periods, and sometimes selected mem- 
bers of the project management will serve as voting members of the Board. 
(For more details of the procedures of proposal evaluation, see NASA 
documents: PR 3.804, parts 1,2, and 3.) See Figure 3 for aa overall view. 

Project Management 

Project teams are assembled from across Goddard *s discipline-oriented 
directorates. Some members are organizationally transferred full-time 
to the project office and are co- located with the Project Manager. in 
these cases, the career advancement of the team member is dependent on 
the Project Manager. Other members are not co-located with the project 
office and are still organizationally assigned to their parent director- 
ate, though they are responsible for specified project tasks. Usually, 
managers at the systems levels and also at the subsystem level for 
projects of high cost, are organizationally assigned to the project 
office. At the component levels, managers remain within their director- 
ates. Final project responsibility is determined according to the pro- 
ject's span across directorates. If the project lies wholly within a 
directorate, final responsibility rests with the head of that director- 
ate. More generally, the project will cross directorates so that final 
responsibility rests with the Program Manager. 
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Extramural projects are monitored and managed by the Goddard 
Projects Directorate. Intramural projects are managed by the Goddard 
Technology Directorate* Projects In any directorate may receive 
additional Support Research Technology fiands allocated by NASA Pro- 
gram Offices (Office for Manned Space Flighty Office for Space Sciences 
arid Applications, etc.) to build long-range capabilities. Separate 
Advanced Research Technology funds are awarded directly to Goddard ^s 
specialty directorates (directorates other than Projects or Technology) . 
There is no formal committment of these long-range funds continuing, 
but Goddard has been able to keep them at a level of about ten percent 
of the center budget* 

Performance Evaluation 

Project evaluations are conducted as an ongoing part of project 
a\;t'.vitles rather than as separate efforts following project comple- 
tion* Practices of frequent commimication and joint decisionmaking 
are stressed rather than formal evaluation procedures. During Phase 
C and D activities, project managers prepare monthly reviews comparing 
the technical performance, cost, and manpower requirements currently 
estimated with the benchmark figures indicated in the Project Plan 
prepared during Phase B* Summaries of these monthly reviews are 
conibined from all projects within a program and are reviewed by the 
Goddard Management Council and NASA Program Administrator* 

ORGANIZATION 

Goddard encourages a steady flow of people between Intramural 
research projects and management of extramural projects* Interaction 
among the two fimctlons is promoted partly by Goddard policies of 
personnel recruitment and management training, and partly by a matrix 
form of organizational structure* The latter emphasizes that Goddard 
does not maintain two separate directorates, one for Intramural acti- 
vity and one for extramural contracts management, each with its own 
permanent full staff* Instead, the mixing of both activities is in- 
duced through the drawing of project personnel from across intramural 
specialties for temporary assignment to a project office* This matrix 
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form of structure is illustrated in the accompanying organizational 
chart of Figure 4. 

Project Managers 

Project Managers arc: chosen from three sources: (1) Studies 
Managers of approved studies plans, (2) Project Managers of termin- 
iating projects, and (3) Goddard management with the greatest tech- 
nical competence in the critical technical areas. The primary criter- 
ion for selection of Project Manager is technical competence; the second 
criterion is management experience. If the Project Manager does not 
have a high degree of competence in the crucial technical areas of the 
project, then his assistant must. 

A Project Manager may or may not be the same person as the Study 
Manager for the project idea. Directors may or may not have formal 
management education and training, but invariable will have had some 
management experience. Many of the project personnel participate in 
Goddard"s management training exercise: GREMEX, a project management 
simulation. Few tour of duty personnel are ever used as Prefect 
Managers • 

Attraction of Intramurel Researchers onto Projects 

Some participation of intramural researchers on project problems 
is financially induced. Goddard provides its Project Managers with 
research funds available for supporting R&D back in the discipline- 
oriented directorates. In this way. Project Manager can encourage 
good intramural scientists to work on problems whose solutions are 
vital to the success of the project by providing funds for those prob- 
lems in particular. In most cases, however, intramural scientists are 
merely assigned to Project Offices. 
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III, PROGRAMMATIC PARADIGMS OF THE NATIONAL CANCER INSTITUTE 
OVERVIEW 

The National Cancer Institute Is one of the divisions of the 
National Institutes of Health, and manages Its extramural grant activ- 
ities largely according to NIH's Diiai Review Paradigm described earlier. 
What makes the Cancer Institute different from the rest of NIH's 
divisions is that besides conducting grants programs to f-^ster fimda- 
mental research , the Institute manages large progranonatlc activities 
targeted toward specific diseases or clinical practices. NCI conducts 
these programmatic activities through various management paradigms 
different from the Dual Review method. The purpose of this section 
is to describe these programmatic paradigms. Before these descrip- 
tions are made, some background and general information is provided. 

In 1937, the National Cancer Institute Act called for the 
establishment of an institute to foster and coordinate cancer related 
research, and in 1944, the Cancer Institute was made a division of the 
National Institutes of Health. In 1971, the Institute operated wl'th 
a staff of approximately fourteen hundred employees and a budget of 
over two hundred thirty million dollars. Of last year's staff, about 
thirty-eight percent were scientific and profes9lonal; twenty-five 
percent were administrative and clerical, and thirty-seven percent 
were technic.il and supporting. Of the budget, approximately fifty 
percent was allocated to extramural grants and training programs, 
ten percent to intramural research, and forty percent to collaborative 
studies. Most of the fifty percent allocated to extramural grants 
was managed according to the conventional NIH Dual Review paradigm. 
Collaborative and Intramural studies constitute the majority of the 
Institute's programmatic activity and are usually managed separately 
from the grants programs. 

The full range of Institute activities consists of basic 
research, administered separately within a Directorate of Extramural 
Activities; practlce-and-service-orlented clinical research, admin- 
istered within a Directorate of General Labs and Clinics; and 
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dlsease-orlented research, administered by the two directorates of 
Etiology and Chemotherapy. The prograrmatio activity of NCI resides 
mainly in the three directorates mentioned last, and all three conduct 
both intramural and collaborative efforts. One Interesting experiment, 
however,* which represents a major departure from traditional, federal 
R&D management practices, is NCI's recent programatio effort targeted 
toward Bladder Cancer and managed extramurally through the extramural 
grants program. The management procedures in this experiment will be 
described along with other practices more normally used by NCI. 

In addition to the three main directorates, the Institute has a 
staff group which participates heavily in developing formal plans for 
major program activities of the Institute. This Office of Program 
Planning and Analysis has developed a method called **The Convergence 
Technique," which is used to formulate rather detailed plans for 
conducting large research programs. So far, this technique has been 
used by the Cancer Institute to develop the plans for six major research 
programs targeted towarc?< special diseases or clinical practices. Details 
of the convergence technique may be found in Appendix B. 

Programmatic activities of NCI encompass three major "research 
cuts" through the problem of developing preventative measures and 
clinical treatments of cancer diseases. These three cuts are illus- 
trated in Fig. 5. 

Figure 5 

NCI's Programmatic Research Attacks on Cancer 

^ R&D Targeted Toward Specific Organs 

Lung Cancer 
Breast Cancer 
Bladder Cancer 
Prostate Cancer 
Pancreas Cancer 



j?<6jD^ Targeted Toward Specific Methodologies 



Virology Chemotherapy Immunology 

formal cells subject to cancer attack 
^lignant cells 

Rehabilitation of treated patients 
R&D Targeted Toward Specific Disease Phases 



-25 



The management of programmatic R&D targeted toward specific methodologies 
resides predominantly in the Directorates of Etiology and Chemotherapy. 
Organ-centered programmatic R&D Is managed within the Directorate of 
General Labs and Clinics, with the one exception of the Bladder Cancer 
Program, to be managed on an experimental basis within the Directorate of 
Extramural Activities* Procedures for the management of each type 
of progranoiatic activity differ in some respects and are common in 
others. These comparislons are outlined below. 

Because the Dual Review paradigm by ^Ich the National Cancer 
Institute conducts most of its extramural grants activity has already 
been described in an earlier report , this section will highlight 
the essential features of the management paradigms used by the Cancer 
Institute in conducting its programmatic activity only. These features 
are: 

For all programmatic R&D activity at NCI: 

o Major ideas for programe arise Intramurally or through 
Interactions between the science and medical communities 
and the NCI advisory groups. 

o Ideas presented to and approved by the National Advisory 
Council on Cancer are developed into formal programs , 
often by a formal planning procedure (The Convergence 
Technique) used to help derive the required R&D content 
to meet program goals* 
For programmatic R&D activity managed intramurally within the 
Directorates of Etilology and Chemotherapy \ 

o A matrix form of program management is used, intersecting 
discipline-oriented organizational "branches** of the Insti- 
tute with special problem-focused organizational ''segments." 

o Segments are the administrative units of NCI collaborative 
research, and segment members Include both Intramural and 
outside scientists. 

o Segment Chairmen^ all intramural scientists, assume 
responsibility for program management and have greatest 
influence on decisions of project selection. 

O 

ERLC 
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o Evaluation of project proposals is conducted through a 

variant of the dual review process using full segments 

and segment chairmen as reviewers, 
o A major emphasis is placed on the directed and integrated 

use of intramural and collaborative research. 
For programmatic R&D activity also managed intramurally ^ but 
within the Directorate of General Labs and Clinics i 

o Task forces are assigned by the National Advisory Council 

to assume responsibility for overall program planning and 

management . 

o These task forces provide the basis for another matrix 
form of program management paralleling that of Etiology 
and Chemotherapy. In General Labs and Clinics, research 
members of NCI discipline-oriented "branches" also serve 
on these programmatic task forces. 

o Task forces include both intramural and outside scientists, 
as do segments. 

o An NCI Steering Committee^ usually composed of the 
intramural members of the task force, has greatest 
influence in decisions of project selection, as do the 
segment chairmen. 

o Evaluation of project proposals is conducted through a 
similar variant of the dual review process using Task 
Force Subcommittees, including both intramural and outside 
scientists, and the Task Force Steering Committee 
reviewers. 

o A similar eiq>hasis is placed on the directed and 

integrated use of intramural and collaborative research. 
For programmatic R&D activity managed extramurally and supported 
by the grants program of the Directorate of Extramural Activities: 
o One large grant is awarded for the management of a 
major programmatic effort. (This is not standard pro- 
cedure for NCI, but is being conducted on an experimental 
basis. ) 
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o The grant will be awarded upon approval by the National 
Advisory Council of a program plan (prepared according 
to the Convergence Technique) presented to the Council. 

o The award will probably be given to one of the intramural 
scientists who participated on the planning cadre which 
prepared the program plan. 

o This grantee wxil then assume all responsibility and authority 
for awarding subgrants on all the R&D activities specified 
in the plan. 

SUMMARY 



General Characteristics 



Primary output: 



Mechanisms of support: 



Managerial emphasis i 



Staffing plan: 



Preventative measures and clinical 
treatments for cancer « 

For all programmatic R&D managed 
intramurally , contracts for studies 
are let on a one-year cycle; each 
contractor must submit a new proposal 
each year even though his project 
covers many years' work. For NCI's 
experiment with extramural management 
of progr aromatic R&D, support is 
provided by one overall grant for 
each program. 

Monitoring of projects receives more 
managerial effort than any other 
management activity, for programs 
managed intramurally. 

Predominantly matrix form. In two 
directorates, problem-focused organ- 
izational segments 9 which are the 
administrative units of collaborative. 
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contracted research, cut across 
dlsclpllne-orlented Institute hranche^^ 
which are the administrative units of 
Intramural research. Practically all 
intramural staff serve on at least one 
problem-focused segment. Intramural 
branch chiefs often also serve as seg- 
ment chairman managing collaborative 
research. In a third directorate, 
similar Institute brandiee Intersect 
with problem-focused task forces which 
are constituted similarly to segments. 

Program Planning 

Sources of new program Ideas for Institute programs come from 
ideas • 

outside peer-directed research, from 
deans o£ medical schools, hospitals, 
the American Cancer Society and other 
cancer-related associations, and mostly 
from current Intramural research. 
All new ideas are presented to the 
National Cancer Advisory Council for 
comments. 

Mechatilsms for planning: If program ideas do not fit into 

current program plans, additional 
plans are prepared often by the Office 
of Program Planning and Analysis, 
together with a director appointed to 
the new program. They use the 
Convergence Technique^ a formal 
planning procedure, to determine inter- 
mediate program objectives, major 
decision points , the necessary projects, 
and data requirements. 
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Coordination: 



Program Development 

Sources of project ideas: 



Means of proposal review: 



Allocation of budget: 

ERLC 



Within each major program, a ''working 
group" of all segment chairmen or 
intramural task force members, depend* 
ing on the directorate, meets regularly 
to review overall progress, reassign 
priorities, and modify any program 
plans as necessary. Across-program 
coordination is the responsibility of 
the NCI Director • 



For R&D programs managed intra- 
murally, ideas for contracts within 
a program are generated both intra- 
mural l.y and from the scientific and 
medical communities. Most project 
contracts are derived logically from 
the data requirements outlined on the 
relevant program plans. 

Contract proposals are usually 
reviewed first for technical excel- 
lence by both intramural and outside 
members of the program management team, 
and second for relevaniJC^ neci^ and 
logical priority by selected intra-- 
mural members of the same teain. in 
two NCI directorates, these gruup.s 
are the program segments and segment 
chairmen, respectively. In a third 
directorate ) they are the Task Force 
Subcommittee and NCI Steering Commit- 
tee , respectively . 

The Institute awards contracts within 
a program from among the set of 
proposals acceptable to the first 
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review group on the grounds of tech- 
nical excellence, and in the order 
determined by the scores of the second 
group, regarding relevance, need, 
and logical priority. 

Monitoring of performance: Monitoring for all contracts is 

conducted by project officers, who 
may oversee several related contracts 
and conduct periodic site visits. 
Quarterly progress reports, and 
annual resubmission of contract 
proposals are required. 

Evaluation of outcomes: No foxnnal procedures of evaluation 

are used. Instead, projects are 
evaluated continually in terms of their 
progress in supplying the data required 
by the overall program plans. 



Ptogram Evaluation 

Mechanism of evaluation: Again, no formal procedures are used. 

Overall program evaluation is measured 
by progress through the program plan. 

ACTIVITIES 



Program Planning 

Programs in the Cancer Institute are both relatively large and 
stable. For example, the three major programs of the Directorate of 
Etiology include: Card ino genes is , with twenty million dollars 
spread across fourteen contracts; Viral Oncology, with forty-four 
million dollars and approximately forty contracts; and Demography, 
with ten million dollars and about eigh,^. contracts. Program 
planning usually is associated with the initiation of a program 
O , revision or addition linked strongly to an already existing program 

and reflective of recent breakthroughs. 
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The specific roles and management procedures for program 
planning used for programs to be managed intrarmdrally differ In many 
ways from those used for programs to be managed extramurally • In 
the former case. If a program revision or addition cannot be Incor- 
porated within existing program plans or Is so large that a whole new 
plan is needed, a program director, to whom responsibility has been 
assigned for the additional work, coordinates with the Office of 
Program Planning and Analysis In generating a new program plan. This 
planning team, often with the help of outside scientists, develops 
program objectives and strategies, a list of initial project areas, 
decides whether or not the projects within an area will be done intra* 
murally or outside, generates requests for proposals, and oversees 
the development of the subprogram throughout its lifetime. The program 
plan, prepared by the planning team and presented to the National 
Advisory Council, is called a Convergenae Chart at NCI because of its 
role in the Convergence Technique. Details of the Convergence Tech-* 
nique are described in the appendix. 

In the case of programs to be managed extramuratty ^ a greater 
participating role is taken by the National Advisory Council. In 
the former case, the Council merely reviews plans initiated intramurally 
and supported by the NCI Director. A major program example of this is 
the Special Virus Cancer Program, managed Intramurally within the 
Directorate of Etiology. Much greater participation by the Council 
was taken in the initiation of the Bladder Cancer Program, managed 
extramurally and supported by the Directorate of Extramural Activities. 

Interest in pursuing a Bladder Cancer Program started within the 
Council in response to high incidence rates and a feeling, supported 
by the American Cancer Society, that a major programmatic effort aimed 
at bladder cancer was feasible at the current time. The Council 
commissioned a task force of intramural and extramural scientists to 
prepare a program plan, and with the help of the Office of Program 
Planning and Analysis, a detailed Convergence Chart was developed. 
The plan was presented to the Council, and if it is approved, one of the 
intramural scientists on the planning task force will probably be 
designated Program Manager and be given a single, large grant to 
O ^ manage the entire program. 

ERIC 
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Whether new NCI programs are to be managed Intramurally or 
extramurally , the program plan, or Convergence Chart, plays a key role 
In defining exactly what R&D activities will be conducted. In addition, 
the chart Illustrates the desired major thrusts o€ R&D efforts and Is 
used both to acquaint outside scientists with current NCI activity 
and to encourage the submission of relevant proposals « As a inanage- 
ment tool, the charts indicate to NCI managers how newly planned activity 
ties in with existing NCI work, what gap areas are left, and where 
program priorities should be placed « Finally, the Convergence Charts 
form the basis for individual work statements and are used as overall 
guides to both project and program evaluation. Because of the 
Importance of these charts, or program plansyto NCI management, formal 
procedures fur their development have been devised and tested. 
Through application of those procedures, plans have been prepared for 
programs in all four NCI director^^tes. Those plans are listed below, 
and an abstract example Is presented in the appendix. 



Program Plan 
(Convergence Chart) 

Special virus cancer 
Chemo-Car cine genesis 
Chemotherapy 
Breast Cancer 
Rectal Colon Cancer 
Bladder Cancer 



NCI Directorates 

Etiology 

Etiology 

Chemotherapy 

General Labs and Clinics 

Extramural 

Extramural 



For additional Information describing the theory and practice of 

the Convergence Technique , see Appendix A. 

Overall differences in the planning procedures for programs 
managed intramurally and extramurally are highlighted in Fig. 6« 
In both cases, the Office of Program Planning and Analysis may contribute 
substantially to the preparation of the program plan. Their 
participation, however, is neither mandatory nor typical for all 
programs. Frequently, the reactions of scientists toward any 
approach which attempts to control or manage science is negative. 
Consequently, the planning staff view their operations as managerial 
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resources for those program managers who wish Co use them. However, 
for many initial skeptics, even brief participation in planning 
exercises has converted them toward favoring the Convergence Technique 
as a useful planning tool, and the concept of planning itself as a 
useful approach in meeting priority objectives. 

Proj^ram Development 

Program additions or revisions are managed in one of two ways. 
In some cases, the constituent projects correspond to components of 
already existing program plans and Institute structures, and are 
developed along with other existing programs. . This means that no 
new problem-oriented Institute segments or task forces are established 
as separate administrative units o£ the new program activity. In 
other cases, new administrative structures are created. Within the 
Directorates of Etiology and Chemotherapy, each of the programs is 
divided Into segments^ typically five to ten segments per program, 
where each segment represents a block of projects similar enough to 
be managed as a total package. Projects are often collected into 
segments with the interests of the assigned segment chairman in mind, 
so that the chairmen tend to remain with the program for its duration. 
Segment chairmen are all intramural scientists, and approximately 
half the members of a segment come from outside the Institute. 

Even within the programmatic R&D activities of the Cancer 
Institute, program development is largely evolutionary. Program 
plans are often rewritten, especially when an area of the program 
starts producing consistently negative results. Redirection of 
effort is then made accordingly. On the positive side, consistently 
negative results within an area often indicate* program progress. 
For example, thirty-two adenal viruses were eliminated as cancer 
agents in the first year of an Institute program. Many of these 
viruses had been postulated for fifteen years as being cancer producing. 

Ideas for projects within a program are generated continually 
both Intramurally, outside, and at project review committee meetings. 
For ideas generated intramurally, the scientist proposing the project 
meets with his branch chief to decide if the project is too big to 
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be worked out by the individual scientist. If it is too big, the 
idea is taken to the segment chairmen. They and the branch chiefs 
then decide if (1) the idea fits into an existing program plan, and 
(2) whether there is sufficient money to support the project. 

If enough money is available, the scientists write an RFP, and 
the same group as above reviews the request prior to its distribution. 
Proposals coming into the Institute in response to the RFP ace 
reviewed by the two types of groups mentioned earlier; the first 
consisting of both intramural and outside scientists, and the second 
consisting of only intramural members. If the second review group, 
rating proposals on relevance, need, and logical priority, cannot 
decide whether or ngt to fund a proposed project, or cannot choose 
between alternative competing projects, they may select an ad hoc 
team to make site visits and make the final choice. This happens 
especially if one of the competing proposals is relatively overpriced. 

Contracts are let in one year cycles. Proposals must indicate 
required levels of support for the current and subsequent years. 
All are resubmitted for review annually. 

The management of funded contracts is done by segments or 
task forces^ depending on whether the program is conducted within 
the Directorates of Etiology and Chemotherapy or General Labs and 
Clinics. Both forms are similarly constituted and have similar roles. 
A single segment chairman or task force member may be running 
several related contracts. Monitoring of performance is done by 
project officers who oversee related contracts and conduct site 
visits. All contracts are visited a minimum of once per year. Some 
get up to twelve visits per year. Typically, contracts range from 
one hundred thousand to two hundred thousand dollars, and a single 
contract officer may handle four or five of them simultaneously. 

Contractors must submit progress reports three times per year. 
The project officer summarizes these reports and circulates the 
summary among the rest of his segment or task force. All projects 
are continually assessed in terms of their progress in providing 
the data required by the program plan. No other forw^l evaluation 
procedures are used. However, a contractor's pverall performance 
Q is usually recorded for reference in ranking future proposals. 

ERIC 
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A description of the program development processes appears in 
Fig. 7. 

The above description focuses on program development within a 
program. Here, procedures for proposal review and contract management 
are relatively straightforward. The more critical problems of program 
development at NCI occur between programs and concern the allocation 
of responsibility for particular program pieces necessary for more 
than one puogram. For example, work on certain viruses may be common 
to the requirements of both the Special Virus Program of the Director- 
ate of Etiology and the Breast Cancer Program of General Labs and 
Clinics. Since both programs operate with relatively separate funds 
and management teams » there may be some conflicts over the allocation 
of work. These problems may even magnify with the inclusion of 
additional programmatic R&D managed extramurally within the grants 
program. Though no formal procedures of resolution have been evolved, 
NCI management has become sensitive to this issue in program development. 

Program Evaluation 

At the program l^vel, evaluation is associated with progress 
through the program plans. No formal evaluation procedures are used. 

ORGANIZATION 

Structure 

The accompanying chai c indicates the four major divisions of 
the Cancer Institute. Etiology > General Labs and Clinics > and 
Chemotherapy do both intramural and collaborative research. The 
Extramural Division does only extramural grants management. 

In the Etiology Division, there are three major programs, each 
headed by an associate director. Each program is further broken 
into branches > each headed by a branch chief. These branches tend 
to be discipline-oriented; e.g.. Biometry, Biology, Chemistry, and 
Experimental Rathology. The branches are the administrative units 
for intramural research activity. Alongside component branches are 
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program segments, each headed by a segment chairman. These segments 
tend to be problems-oriented; e.g.. Solid Tumor Viruses, Immunology, 
Molecular Carcinogenesis, Bioassay, Lung and Information and Research. 
The segments are the administrative units for collaborative (con- 
tracted) research within the Directorates of Etiology and Chemotherapy. 
A segment is composed of between five and ten people serving as 
program officers for a few contracts. In general, these members 
perform two roles: as intramural researchers within a branch, and 
as program officers within a segment. Segment chairmen commonly play 
double roles, serving as branch chiefs as well. 

Within Etiology, the Carcinogenesis program has eight segments, 
the Viral Oncology program has nine segments, and the Demography 
program has seven. A few segments are nearly vertical, drawing most 
of their members from a single branch. Most are horizontal, cutting 
across many branches* Ninety percent of the intramural staff serves 
on at least one segment. 

A similar matrix form of program management exists in the 
Directorate of General Labs and Clinics. Rather than being called 
segments, the problem-focused administrative units of collaborative 
research are called task forces, and are headed by a Task Force 
Steering Committee of intramural members. Intramural task force 
members similarly serve^also in discipline-oriented branches of 
General Labs aad Clinics. Inc3.uded are branches of surgery, radiology, 
immunology, endicrinology , dendatology, and metabolics. 

The Cancer Institute tries to maintain organizational stability 
on one hand, and on the other, to be responsive through organizational 
changes, to major breakthroughs. Most organizational changes are 
incremental, though some have been major. For example, the Etiology 
Division did not exist in 1965, yet now has three major programs 
within it. Also, if the idea of host change as a means of cancer 
prevention and cure gains Institute support, immunology may gain 
divisional status as well. 



Staffing Policies 

Program managers almost always come from a research background, 
and get some staff experience before assuming program responsibility. 
Earlier institute experiences with trying to train good administrators 
to be good science managers failed because of their lack of technical 
background. Typically, each program manager handles projects whose 
budgets total between thirty-five and forty million dollars. 
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IV, AIR FORCE MANAGEMENT PARADIGM FOR WEAPONS SYSTEM DEVELOPMENT 
OVERVIEW 

Within Air Force programs concerned with overall strategic and 
tactical forces are many projects of weapon system development. These 
projects are often so large, in budget and complexity, as to require 
their own systems for planning, contract selection, and contract 
management. Much of the management activities of the Air Force focuses 
down at this project level. Temporary management organizations and 
often procedures themselves are project-specific. For these reasons, 
this section dealing with Air Force R&D management methods will empha- 
size project initiation, contractor selection, and contract management, 
and will focus on projects of major weapon systems development.* 

Procedures for all military system development and procurement, 
more than procedures for R&D management in non-defense sectors, reflect 
underlying processes of advocacy; with profit-making developers compet- 
ing for contracts from the services, and the services requesting funding 
from Congress. Consequently, the management of these military R&D 
projects emphasizes balancing among needs and interests, and controlling 
potentially overzealous developers. This differs greatly from the 
situation in which R&D projects must be stimulated in areas of need, 
which is the prime management task in some of the other R&D agencies 
discussed earlier. 

Decisions first by private contractors, concerning which unsolicited 
project ideas to promote and to which RFPs to respond, and second, by 
the services, concerning which proposed systems merit Air Force advocacy 
for funding, are influenced by a wide variety of intelligence systems, 
sensitivities to technological advances, clues of Congressional attitude, 



A major project is defined as one having either an estimated 
RDT&E cost in excess of $50 million, or an estimated production 
cost in excess of $200 million. 
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and formal military planning documents. The management of system 
development reflects this sensitivity as to what mij^t be approved » 
as well as strict compliance to a large number of formal procedures 
for initiating, directing, approving, monitoring, and controlling 
major extramural projects. 

A detailed and complete description of the entire process of 
program planning and development for any of the armed services would 
require volumes. Instead, this section attempts to convey the broad 
features of the managerial procedures used specifically by the Air 
Force throughout most of the last decade in developing major systems. 
It highlights both project initiation and control. More formal and 
complex descriptions of the flow of decisions and documentation in 
this management process may be found in the instructions and guides 
listed in the past Air Force Regulations concerning system development, 
(see AFR-57-1, APR 375 series, and DOD 4100.35) 

In July, 1971, the Air Force issued a new regulation for program 
management (APR 800--2) which supercedes much of the procedural require- 
ments used in the past, and attempts to reduce formal management and 
review activities by delegating more responsibility to the final program 
user (implementing command) and to the designated Program Manager. Details 
of this regulation have just begun to evolve. Consequently, this section 
will describe those procedures used over the last 8 years, and will con- 
clude with highlights of the new procedures implied by AFR 800-2. 

In the past. Air Force project development consisted of three 
phases, each ending with a requirement of approval by the Office of 
the Secretary of Defense (DSD) for continuation. The Conceptual Phase 
traced the system from its perceived need in any of the Air Force 
operating divisions, stated as a Required Operational Capability or 
ROC; through review panels and supplementary studies administered by 
Air Force Headquarters; to OSD approval of the technical development 
plan for the project. The Validation Phaee Included the establishment 
of a Systems Project Office (SPO) to manage the proposed development; 
additional trade-off studies, technical feasibility studies, and miss- 
ion analyses as required; the development of RFPs; and OSD approval 
for development. Finally, the Development Phaae consisted of contracted 
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development under close management of the SPO according to detailed Air 
Force Regulations. 

For this last phase, a System Project Office (SPO) was established 
as a temporary organiz&tion, existing only duri.ng the development life- 
time of the project. Once the system was produced and operational, 
this organization was disbanded. Even within the temporary SPOs, 
turnover of managerial personnel was high, particularly among the 
military participating as part of their tour of duty« SPO size ranged 
from about fifty, as in the contemporary Maverick and SRAM missile 
projects, to as high as four hundred fifty, as in the Mihuteman missile 
project. 

The essential features of the former management paradigm for Air 
Force system development projects were: 

o Major project ideas were initiated from any of the Air Force 
operating divisions, such as the Strategic Air Command, 
Tactical Air Command, and Systems Command, and were stated as 
a "perceived military need" or Required Operational Capabil- 
ity (ROC). (No change under AFR 800-2) 

o The development and evaluation of project ideas (ROCs) were 
managed within Air Force Headquarters by area-focused panels 
of officers prior to any position of Air Force advocacy. for 
funding. (No change under AFR 80C-2) 

o Approval for each individual project had to be obtained from 
Air Force Headquarters, the Office of the Secretary of Defense 
and Congress . (No change under AFR 800-2) 

o All major development projects were conducted extramurally ^ 
under close management of a System Project Office (SPO) of 
finite life and subject to detailed formalized management 
procedures. (Under AFR 800-2, SPOs are eliminated and a new 
emphasis placed on single Program Managers and a reduction of 
formalized procedures.) 

o Major managerial emphasis is placed first on selection and 
justification of project ideas prior to positions of advocacy 
and preparation of any RFP, and second, on control of extra- 
mural project development subsequent to contract award. (No 
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change under AFR 800-2) 
o Contract proposals are reviewed and selected by temporary, 
formal Source Selection Boards^ established specifically for 
each major project, and awards are made to industrial organi- 
zations rather than individual scientists. (Under AFR 800-2, 
most of the cumbersome features of formal contract selection 
are eliminated, and the number of people participating is 
greatly reduced. See details in the text.) 



SUMMARY 



General Characteristics 



Relevant project output: 



Mechanism of support: 



Staffing plan: 



Development of major weapon 
systems • 

Detailed development contracts 
awarded to outside private aero- 
space organizations. Contracts 
may be fixed fee, cost plus fee> 
or incentive type contracts. 

Following contract award,, tempor- 
ary System Project Offices were 
established inhouse to direct the 
overall management of the develop- 
ment projects. (Undt^r AFR 800-2, 
these large SPOs will be replaced 
by much smaller Program Offices 
assisting the Program Manager.) 
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Program Planning 

Sources of new 
program ideas: 



Sources for new program ideas in- 
volving projects of weapon system 
development may be any of the 
many intelligence documents utilized 
by Air Force or the many formal 
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Mechanisms for planning: 
Coordination: 



Program Development 

Sources of new 
project ideas: 



Development of project 
ideas : 



planning documents produced by 
them. 

Formal planning documents are 
prepared by the Joint Chief of 
Staffs by Air Force Headquarters » 
and by Systems Command. 

Coordination among project plans 
is promoted by frequent meetings 
of the Air Force Council made up 
of the Deputy Chiefs of Staff of 
all the functionally oriented 
divisions, and of the Air Staff 
Board made up of the heads of the 
departments under the Deputy Chiefs 
of Staff. A major concern of these 
groups is in determining the 
set of potential projects which 
can best meet the needs of Air * 
Force programs. 



Project ideas may arise fro^ any 
general officer in any of the 
operating divisions such as Stra- 
tegic Air Command or Systems 
Command, or may be suggested by 
any potential contractor. 

A project idea, stated as a 
quired Operational Capability 
(ROC) is first coordinated with 
all other divisions to prevent 



Preparation of the RFP: 



Selection of external 
contractors: 



duplication and foster joint 
V'iuefits. It is then reviewed 
t y the Air Force Council and by an 
inhouse area-focused panel main- 
Lalned continuously by the Deputy 
Chief of Staff for RU). Follow- 
ing subsequent studies required 
by the panel and the development 
of a Concept Formulation Package/ 
"^achnical Development Plan (CFP/ 
TDP) the project idea was reviewed 
by the Air Staff Board, again by 
the Air Force Council, and finally 
by OSD and Congress. (Under AFR 
800-2, the CFP is replaced by the 
document^ already prepared during 
program advocacy, and the TDP and 
many other formal documents are 
replaced by the single Program 
Management Plan.) 

Following approval for funding, 
a Systems Project Office (SPG) 
was established within Systems 
Cononand to prepare the RFP for 
the project contract. (Under 
AFR 800-2, SPOs are eliminated 
and responsibility is delegated 
to a single Program Manager.) 

Proposals were reviewed and con- 
tracts were awarded by formal 
Source Selection Boards estab- 
lished solely to review proposals 
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Monitoring p£ performance: 



Mechanism of project 
evaluation: 



ACTIVITIES 



for an individual project. Each 
Board h^id Its own advisory council 
which determined criteria for 
selection, and its o«m set of 
technical committees to score pro- 
posals in their particular area* 
All w^e disbanded following con- 
tract award. (Under AFR 800«-2» 
the Board and technical committees 
are all replaced by a small single 
committee of 10-15 personnel.) 

Contractor performance in the 
development of a new system was 
monitored closely by the System 
Project Office throughout develop- 
ment. The SFO was assisted in 
monitoring contractor compliance 
to Air Force regulations by an 
outside Air Force Plant Repre- 
sentative Office. (Elimination 
of SPOs under APR 800-2.) 

No formal procedures are used 
within the Air Force. However » 
other government, agencies such as 
the Government Accounting Office 
(GAO) often conduct unsolicited 
project evaluations in situations 
of potential underachievement in 
performance y cost, or time. 
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Within Air Force Headquarters » project management activities for 
weapon system development are directed largely toward initiation of 
potential projects for submission to Congress for funding. Subsequent 
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to funding approval, major managerial efforts are directed to contractor 
selection procedures. Because many Industrial organizations often compete 
for Air Force project contracts, and because the amount of money In any 
contract is often quite high, detailed and formal procedures must be 
followed by temporary project-specific selection groups in awarding 
contracts fairly, and in accordance with pre-defined standards. Follow- 
ing contract award, project management activities emphasize control, 
and in the past, responsibility was directed to temporary , project- 
specific System Project Offices for constant monitoring and communica- 
tion with the contractors selected. 
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Initiation of Potential Projects 

The perceived need giving rise to a major project idea may originate 

from a number of intelligence and planning sources. Some are the logi- 

cal consequences of new threats. Others are spawned from technological 

breakthroughs permitting new military capabilities. Most are products 

of the continual interchange of ideas between the Air Force with its 

perception of its own strengths and weaknesses, and Industry and its 

perceptions of possible new products and capability advances. 

On the military side, commanders of the Air Force operating 

divisions receive continual intelligence reports about the capabilities 

of potential adversaries and are provided with a host of planning 

documents indicating anticipated events and desired future capabilities. 

Some of the formal , annually revised^ planning documents related to 

the needs and desires of new systems are listed below: 

Prepared by the Joint Chiefs of Staff: 

JIEP — Joint Intelligence Estimate for Planning 

JSCP — Joint Strategic Capabilities Plan 

(1 fiscal year into the future) 

JLKSS - Joint Long-Range Strategic Study 

(10-20 years into the future) 

JSOP ~ Joint Strategic Objectives Plan 

( 2-10 years into the future) 0^"^ 

JSDOD — Joint Research and Development Document 

(Broad capabilities desired for systems and material 
in period covered by JSOP, and technological accomr 
^ . piishments desired for period of JLRSS) 
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Prepared by Air Force Headquarters: 

USAF Planning Concepts — Conceptual foundations for 

desired capabilities. Revised annually by the 
Directorate of Doctrine, Concepts, and Objec- 
tives, under the Deputy Chief of Staff for R&D. 

Technological Horizons Document — Revised annually by the 
Directorate of Operations Requirements and Develop- 
ment Plans, under the same Deputy Chief of Staff. 

Prepared by the Air Force Operating Commands: 

Technological Objectives Plan — Prepared by Air Force 
Systems Command. 

The initiation of a specific proposed project, stated as a Required 
Operational Capability (ROC) , involves a complex procedure of coordina- 
tion and multiple review, includes the generation of many supplementary 
studies of feasibility and justification, and culuij.nate^ in the production 
of a Concept Formulation Package/Technical Development Plan (CFP/TDP) 
which outlined the entire system design, management plan, and time 
schedule of development activities. (This document and others, have 
been replaced by a single Program Management Plan«) An abstracted, 
pictorial description of this process is presented in Appendix C. Only 
the relevant portions of the Air Force hierarchy are shown« Each page 
illustrates a single step in the generalized procedures of project 
planning. The procedures illustrated end with the preparation of the 
project plan, conducted extramurally and under the overall management 
of the System Project Offices formerly used by the Air Force • 

Contractor Selection 

In the past, for each RFP, a Source Selection Authority and a Source 
Selection Evaluation Board were specially appointed by the Secretary 
of the Air Force, and included representatives of Systems Command, 
Logistics Command, and the system's using command. The Board usually 
had about five to ten generals and perhaps a few senior civilians such 
as the Assistant Secretary. A separate Source Selection Advisory Coun- 
cil was set up to determine the criteria to be used in rating proposals. 
In addition, each Board maintained five committees to rank proposal 
independently of each oth^3r on their particular aspect. These included 
a Technical Committee (responsible for assessments of engineering 
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capabllitles and plans) , a Management and Production Comnlttee (assess- 
ing production capabilities), a Logistics Committee, a Cost Committeei 
and a Cost /Effectiveness Committee. Committees had about twenty- 
five members each, with technical backgrounds appropriate to their 
aspect of the project. 

The evaluation techniques used by the Air Force were designed to 
prevent presentation to the Source Selection Authority of reports which 
simply compared one proposal to another. Instead, the techniques 
emphasized the development of a set of standards prior to proposal 
review and against which companies* approaches could be measured. For 
each of the committees mentioned above, the Advisory Council prepared a 
list of iteme^ fairly broad in scope, for which an evaluation was made 
on all proposals. An example of an item involving development of an 
aircraft is **Cabin Environment." Each of the committees then subdivided 
items further into factors. A factor under the item Cabin Environment 
is "Soundproofing." Standards were then prepared for each factor. 
Generally, standards did not exceed that which was specified as minimally 
acceptable in the RFP and did not address subject matter not specified 
in the RFP. 

Each committee indicated for their factors whether the companies' 
proposals exceeded minimum requirements » met minimum requirements, or 
failed to meet minimum requirements. They then developed an overall 
score for each item based on the component factor rankings. Item scores 
were from 0 as unacceptable to 10 as exceptional. If a company's pro- 
posals met all requirements within an item, but did not exceed them, the 
item was ranked 5. If the company's proposal offered seme unique 
device, process or approach which, for example, saved time, material, 
or reduced risk, then it was scored 6,7, or 8. If the company showed 
evidence of a rare solution which was exceptional in all aspects, then 
it was scored 9 or 10 on that item. Failure to meet minimum require- 
ments of an item was scored 4 through 1 depending on the importance 
of the deficiency, what must be done to correct it, and the impact the 
correction will have. 

*"The Source Selection Process" - a Preliminary Draft ASD Manual, 
15 June 1969. 
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We. i^ts applied to Indicate the importance of the different items 
in coming up with an overall evaluation were determined by the Selection 
Advisory Council and were not made known to the evaluators serving on 
the committees during the evaluation period. Consequently, individual 
evaluators could not determine during the course of the evaluation which 
of the companies achieved the highest overall ranking. 

Following item scoring, committee heads coordinated their findings 
so that the inter-item interfaces and relationships would be fully 
explored and that deficiences Ibund in the technical evaluations were made 
known to the Cost and Cost/Effectiveness committees. 

An ad hoc super-committee then applied the item weightings deter- 
mined by the Advisory Council, rank ordered all proposals on the basis 
of an overall score, and briefed the Source Selection Evaluation Board 
on the relative rankings. Decisions of contract award were then made 
by the Board and reviewed by the Secretary of the Air Force. (For 
further details of this past source selection procedures, see DOD 
3;i00.9, DOD 4105.62, AFR 80-20, and AFR70-150 

Under the new Air Force regulation (AFR 800-2), most of this pro- 
cess has changed. The new procedures call for only a single committee 
of ten to fifteen people to advise the Source Selection Authority on 
contract proposals. This represents a major policy change, the im- 
plications of which are still unknown. Though past procedures were more 
cumbersome and costly, they did provide effective means for fair, 
comprehensive review of highly complex programs. 

Contract Management 

Under the former system, once a contract had been awarded, the full 
responsibility for the management of the project was given to the System 
Project Office. This office maintained constant contact with the contrac- 
tor (either a single prime contractor who subcontracted for required com- 
ponents and systems, or a number of associate contractors each respon- 
sible for a major portion of the project) and participated in some of the 
technical decision making. The SPO received monthly formal progress 
reports and cost performance reports from the contractor, and maintained 
surveillance for compliance with Air Force regulations with the assistance 
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of an Air force Plant Representative Office at the contractor's plant. 

The managerial emphasis of the SPO was on controlling the contractor 
to keep abreast of the time schedule cf development activities as 
detailed in the Technical Development Plan, to stay within the approp- 
riated budgets, and to meet the development deadline for initial 
operating capability. SPO engineers maintained continuous contact with 
contractor engineers for these purposes. At the conclusion of fligiht 
testing and production, the SPO was disbanded. Responsibility for 
the maintenance of operating systeni^^ was then assumed by the appropriate 
Air Material Area (AMA) under the jurisdiction of Air Force Logistics 
Command . 

A description of the entire process of project planning and develop- 
ment under the former Air Force system is provided in Figure 9 . 

FORMER ORGANIZATION 

Structural Relationships 

A typical organization structure for System Project Offices which 
existed as part of the Air Force system for program development over 
the last 8 years is ghown on page 54 . The organization of the entire 
Air Force Department is abstracted in the figures of Appendix C. None of 
this laLLer structure has been changed by th« new regulation for pro- 
gram management. 

Project Directors and Staff 

Most SPOs contained mixes of military and civilian personnel. 
Turnover among System Project Directors and among their staffs had 
sometimes led to confusion in the management of development projects, 
and efforts toward extending the tour-of-duty of project directors and 
providing more civilians in the SPO were being taken by the Air Force. 

Project directors were almost always full colonels, and had exten- 
sive backgrounds in procurement management. They generally had advanced 
degrees, but not necessarily in technical fields. Many were graduates 
of Air Force Schools and so had some formal training in engineering. 
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Figure 10: STANDARD STRUCTURE OF THE SYSTEM PROJECT OFFICE 
UNDER THE FORMER REGULATIONS FOR PROGRAM MANAGEMENT 



SYSTEM PROJECT DIRECTOR! 



CONFIGURATION i 
MANAGEMENT ! 
(Staff role) 



PROGRAM CONTROL | 

Budget responsibility j* 
for the program; i 
prepares cost reports | 
to higher headquarters? 



ENGINEERING 

{Technical responsibility for 
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User Representation within the SPO 

Included within the Systems Project Office responsible for the 
development of new Air Force projects under the old system, vere 
representatives of the training and U:»ing Commands who were respon- 
sible for project implementation and utilization following development. 
Early participation by users was included to minimize unanticipated 
obstacles to project use later on, and ensure development of systems 
containing features desired by project users. 

This last feature became one of the fundamental principles of 
program management under the new regulation issued in July, 1971. The 
concept of user participation in development was extended to user 
responsibility and control. Headquarters participation was reduced 
and the concept of a large System Project Office was supplanted by the 
concept of a single Program Manager appointed by the implementing 
command rather than by Systems Command. 

Additional changes in program management initiated by APR 800-2 are 
described below. 

NEW PROGRAM MANAGEMENT PROCEDURES EFFECTIVE 27 JULY 1971 UMDER APR 800-2 

The following description highlights and paraphrases the new 
procedures designated in AFR 800-2. Where appropriate, major changes 
are indicated. 

General Philosophy 

The regulation delegates maximum authority and responsibility to 
the implementing command and the designated Program Manager for the 
conduct of a program within approved performance, schedule, and 
funding parameters. Decentralized management principles are emphasized 
and the single manager concept is to be employed to the extent practicable. 

Allocation of Responsibility 
a. USAF Headquarters: 

(1) Establishes and verifies requirements. 

(2) Conducts program advocacy. 

O 
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(3) Issues program management directives (PMDs) which: 

(a) Initiate, approve, change, modify, or terminate 
programs . 

(b) Designate the implementing command for programs, 
define the task, and delegate the program manage- 
ment task to that command. 

(c) Designate participating commands and their respon- 
sibilities. 

The Implementing Command: 

(1) Is responsible for the program task as defined in the PMD. 
(This Is a major change. Formerly, program development 
was the responsibility of Air Force Systems Command, 

and participation of the implementing command was limited 
to representation in the SPO.) 

(2) Appoints the Program Manager and appropriately staffs 
a program office. For major programs, this should be 
sufficiently early in the conceptual phase to allow the 
Program Manager to participate in program advocacy. 
(Again, a major change. Previously, Program Managers 
(SPO Directors) were appointed by Systems Command.) 

(3) Delegates maximum authority and responsibility to the 
Program Manager. 

The Program Manager: 

(1) Organizes, plans, directs, and controls the program, 
utilizing the advice and recommendations of the parti-- 
cipating organizations. 

(2) Tailors the organization of the program office and the 
selection and application of management systems to the 
needs of the particular program within the constraints 
specified by the PMD and implementing coomand supplements. 

(3) Makes technical and business management decisions within 
the approved program to accomplish program objectives. 

(4) Establishes the need, scope, costs, and schedule for all 
program related effort. 

(5) Assesses and documents the impact of proposed changes 
which alter approved performance, schedule, and cost 
objectives. 

(6) Prepares and issues a program management plan (PMP) 
in consonance with the PMD and implementing command 
supplements. The PMP is tailored to the needs of 
the program and will not require higher headquarters 
approval unless such approval is specifically required 
in the PMD. 



(7) 



Maintains a continuous assessment of his program's 
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progress and performance versui^ requirements, threat, 
schedule, and costs, and informs higher headquarters 
of recommended changes. 

Commun i ca t ions 

Program Managers are charged with promptly reporting appropriate 
problems to the proper level for timely resolution. To do so, a 
direct channel of communications, called the BLUE LINE is applied to 
programs specified by USAF headquarters. This provides direct commun- 
ication from the Program Manager to the Commander (implementing 
conmand) , Chief of Staff, and the Secretary of the Air Force. The 
participating personnel included in this BLUE LINE channel are indicated 
in the following illustration. (Again, a major change. Under former 
procedureSf communications between a System Project Director and AF 
Headquarters were severely restricted, each successively high layer 
requiring detailed, formal review before critical information could 
be passed up the chain of command.) 

Overall, one can see a major change in Air Force program management 
emphasis toward decentralization, reduced formality in procedure, and 
redirection of development control to the Implementing command. Actual 
effects of these policy changes have yet to be realized. 

AIR FORCE BLUE LINE CHANNEL OF COMMUNICATION FOR PROGRAM MANAGEMENT 



Secretary of the Air Force 

I 

Chief of Staff 

I f ■' — — Staff 

— Commander Implementing Command 

- Staff 



Commander Participating Commands 

. -Staff 



I Program Program Program 

I Man a ge r Man a ge r Manage r 



-58- 



V> QUANTITATIVE MQMILS FOR PROJECT SELECTION 
INTRODUCTORY VERSE: 
Title: 

''The Company President's Completely Perfect and 
Absolutely Quantitative Method of Evaluating His 
R&D Program" 

by Ned Landon 

GE's Corporate R&D Center 

I multiply your projects by the words I can't pronounce, 
And weigh your published papers to the nearest half an ounce; 
I add a big fat bonus for research that's really pure. 
(And if it's also useful, your job will be secure!) 

I integrate your patent rate upon a monthly basis; 

Compute just what your place in the race to conquer space is; 

Your scientific stature I assay upon some scales 

Whose final calibration is the Company net-to-sales • 

And thus I create numbers where there were none before; 

I have lots of facts and figures — and formulae galore — 

And these quantitative studies make the whole thing crystal clear; 

Our research should cost exactly what we've budgeted this year. 

OVERVIEW 

In this final section on Models for R&D Project Selection, the 
style of presentation changes from one of detailed description of 
management methods by sample agencies to one of general overview and 
current state-of-the-art. This change occurs primarily because of 
the great diversity among government agencies and industries in their 
use of quantitative models for project -selection. To examine only a 
few cases and leave the impression that this typifies the formulation, 
use, and reliance on quantitative models would be less accurate than 
a general overview of the kinds of models existing, the overall 
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advantages and requirements of model use, and the general attitude 
of uses toward the appropriate role of models in R&D decisionmaking. 
Consequently, this final section adopts the latter strategy and 
uses as its information source the published literature on the subject 
rather than interviews with members of selected agencies. 

This overview deals with five managerial questions concerning quan- 
titative models for R&D project selection: 

(1) Are Project Selection Models totally quantitative, as 
the introductory verse implies? 

(2) In general, what are the advantages and limitations of 
using quantitative models for project selection? 

(3) In practice, are project selection decisions actually made 
by computerized models , or do the models serve only as one source of 
input to human decisionmakers? 

(4) Are computerized R&D Project Selection Models wide.«^pread 
among industries and government agencies? 

(5) What are some examples of different Project 
Selection Models? 

As a management paradigm. Project Selection Models focus on 
only one aspect of a highly diverse management process. This differs 

considerably from the three agency management paradigms presented 
earlier* Consequently, the content of this final section should not 
be construed as an alternative approach to total management, but rather 
as a supplementary management tool, applicable in concert with each 
of the agency management paradigms described earlier. 

In general j .the essential features and focus of Project Selection 
Models are: 

° In preparing information inputs for model use, groups of 

technitxd' advisors provide quantitative assessments according 
to specified criteria, for each project or research area. 

^ Assessments are combined according to some mathematical model 
by which projects or areas may be compared or budgets allocated. 

° Recommended budget allocations are made all at onae according 
to the current model solution, usually in relation to a 
fixed budget cycle. 
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SUMMAKY 

General Characteristics 
Primary output: 

Mechanisms of support: 
Managerial emphasis: 

Staffing plan: 

Coordination: 



Sources of project 
ideas: 



Allocation of 
budget: 



Evaluation: 



Allocation of fixed budget to 
candidate projects or project 
areas • 

Not relevant to paradigm. 

Program planning only. Monitoring 
and control are not considered. 

Requirement of sources of 
technical assessments. 

Coordination In project planning 
is provided by some models in 
terms of systematic consideration 
of project Interactions. Other 
models provide no consideration 
for coordination. 

Provided by certain models through 
the identification of technological 
areas In which R&D is desired. 
Considered by most ir-odels as 
given inputs. 

Determined, in part, by application 
of the specific model used. Some 
review and modification according 
to managerial judgment almost 
always occurs. 

Some evaluation occurs ac a 
by product of recommendations for 
project renewal. Even here, the 
Intent of model application is 
strictly toward what to do next 
rather than how well have we done. 
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CRITICAL ISSUES 

Are Project Selection Models Absolutely Quantitative^ as the 
Introductory Verse Implies? 

If by "absolutely quantitative" one means uninfluenced by human 
Judgment, the answer to this question Is definitely no. Because of 
the uncertainties regarding technical success, time and cost, effective- 
ness and value, all project selection models use human judgment as their 
basic input. These judgments are, however, expressed in terms of 
numerical values. Accordingly, the models using them are quantitative. 
But placing numbers on judgments does not change the fact that judgments 
are the fundamental ingredient, and whether expressed as numbers or 
sentences, are still judgments. 

Why then bother with translating judgments into numbers at all? 
With Judgments expressed in numerical values, one can formalize a 
set of decision criteria and assign numerical values to their relative 
importance. These weights can then be ccpplied consistently to a large 
number of potential candidate projects. When projects must be compared on 
the basis of multiple criteria and when the tating of each project along 
each criterion is complex, it is extremely difficult if not impossible 
to treat all projects consistently unless some numerical characteriza- 
tion is used in place of lengthy statements of judgment. 

In General, What are the Apparent and Actual Advantages and Limitations 
of Using Quantitative Models for Project Selection? 

If quantitative models for project selection are 5? till only 
applications of human judgment, what are the apparent attractions of 
models to some R&D managers, and what are the actual advantages and 
limitations that result from their use? In answer to the first part, 
the idea that there Is a quantitative basis for a difficult managerial 
task Is generally appealing. In most R&D organizations, the setting for 
project selection decisions is as follows: 

(1) The organization has an annual cycle for project selection 
and budgeting. 

ERIC 
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(2) Competition exists for support among several candidate pro jects . 

(3) The total budget for the projects is less than the sum of the 
candidate project budgets. 

To economists and management scientists » this is an example of a 
classical resource allocation problem of the kind for which quantita- 
tive solutions have been developed. However, the quality of any mathe- 
matical solution is no better than the quality of the input provided, 
and in practically all R&D scoring models currently available, the 
quality of at least half the input is severely limited. The more 
certain, and predictable half is that concerned V7ith project cost, 
although even here large uncertainties are present. But few projects 
are selected on the basis of cost alone. Most selection decisions are 
concerned with comparisons of: 

o Cost and Economic Benefit: Project costs are compared to an 
economic valuation of all the benefits (direct and sometimes 
also indirect) derived from the project, 
o Cost and Effectiveness : Project costs are compared to an 
index of project impact or effectiveness in achieving 
specified objectives. Effectiveness itself is a multidimen- 
sional characteristic. The comparisons below indicate specific 
components of an effectiveness measure, 
o Cost and Relevance: Project costs are compared to ^^n index 
of relevance toward a specified objective or program field, 
0 Cost and Estimated Success: Project costs are compared to 

estimates of the probability of technical success. 
0 Cost and Progress: Project costs are compared to an index 
of how far the success of the project would move the patent 
program through a series of milestones, 
o Combinations of these factors. 
Estimation of the second half of these inputs is subject to far more 
uncertainty than estimates of costs. 

In the selection models, measures of benefit, effectiveness, relevance 
probability of success, consequent progress, or similar criteria are 
usually made by groups of technical advisors through discussions and debate 
They proceed in the same way as the peer panels of other agencies who 
decide directly on project selection. The difference between selection 
models and direct panel review is that panel reviews provide 



judgments of each project la all Its aspects In competition with other 
projects, while the technical groups \ised in project selection models 
provide judgments only of each aspect of a project, one at a time, 
separate from both other aspects of the same project and from other 
projects. Judgments in this latter approach are combined through one of 
the selection models to form a comprehensive assessment of a specific 
project and a subsequent comparison among all projects. 

The selection model's advantages cf additional objectivity through 
explicit treatment of multiple criteria and consistent combination of 
technical assessments must be balanced against the user's perceptions of 
how completely and how accurately the model's rules conform to his own 
desired selection criteria. Frequently, subtle indications of the quality 
of the investigator or his Institution, or complex issues of potential 
inter-project relationships and balance among all the projects in an 
organization's R&D portfolio do not enter into a model, yet equally 
frequently, they are quite Important to the decisionmaker. 

In Practice, Are Project Selection Decisions Actually Made by Computerized 
Models Without Additional Human Intervention? 

Throughout the literature on project selection models, concerning 
both methodology and usage, is the impression that the results of model 
application serve as input to human review of budget allocation. Often, 
reference is made to model use in producing alternative sets of candi- 
date projects which satisfy different criterion weightings of import- 
ance. For example, the models may be used to answer questions such as: 
What if we valued this criterion more than this one? Or what if v/e 
decreased the total budget available by ten percent? Nowhere in the 
literature did we encounter a situation described as one in which the 
selections are made directly from application of a decision model 
without any subsequent review by committee or manager. Most descriptions 
of use emphasize the model recommendations as a basis for further 
deliberation. 

At a recent conference on the administration of research, one company 
president was quoted as saying: "My job begins where the numbers leave 
off ...(but) I prefer to leave off from good numbers!" The numbers 
provided by quantitative models are "good" only in that they provide 
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the final decisionmaker with a fairly consistent application of the 
same criteria to all candidate projects and A consistent comtination 
of all assessments to facilitate comparisons among projects. Whether 
or not the numbers are *'good" in terms of the technical assessments 
of measures of benefit, relevance, or whatever, is independent of the 
quantitative model and dependent instead on the competence and degree 
of effort given by the staff in preparing inputs for the model. 

Are Computerized R&D Project Selection Models Widespread Among 
Industry and Government Agencies? 

Surveys on the use of quantitative models in practice conducted 
within the last decade have concluded that though interest in project 
selection models have been high, and the proliferation of various model 
has been following an exponential growth curve, the actual use of model 
in practice has been both infrequent and limited in scope. The first 
major survey published in 1964 by Baker and Pound (6) stated: 

Although the literature, interviews, and seminar data 
are not conclusive on the matter, it does suggest that 
there has been little thorough testing and only scattered 
use of the proposed methods. 
The authors do mention, however, that earlier surveys by Harrel and 
Quinn show that "many laboratories use some sort of quantitative 
technique part of the time." 

In late 1966, Albert Rubenstein (7) published a second survey of 
theory and practice of R&D evaluation which concluded: 

The practice of project selection in industry and govern- 
ment is dominated by ... methods depending heavily upon 
individual or group judgment and using very little 
quantitative analysis. The use of cost and return esti- 
mates is common, but very few organizations employ any 
formal mathematical model for combining these estimates 
and generating optimal project portfolios* 
Rubenstein continues: 

In following the field closely since 1950, one detects very 
little increase in the use by operating R&D organizations of 
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the more qxiantltatlve methods There are some clearly 
evident reasons for this; the complexity of many of the for- 
mulations, the requirements for data that is not generally 
available, the omission from some formulations in dealing 
with the diverse kinds of selection decisions that arise 
in R&D. 

No indications in the literature exist that widespread use of quanti- 
tative models has occurred subsequent to these surveys, o^: that the 
problems described a& reasons for their lack of general ui^e have been 
solved. 

What Are the Underlying Princij>les in Different Project Selection Models? 

All quantitative models for project selection are derived from 
some or all of three basic concepts: 

(1) The concept of using "relative weightings of importance'* 

in combining assessments among different dimensions or criteria. For 
example, a manager may indicate that measures of the probability of 
success are twice as important as measures of the expected time to 
completion and should therefore receive twice as much '^weight" in 
determining whether one project should be selected over another. 

(2) The concept of "discounting" streams of income and expense 
to produce comparable valuations of money flows. This procedure 
indicates to a manager that in situations of excess funds, cash at the 
current time may be more valuable than the same amount of cash at a later 
time because of the opportunity of earning interest in alternative 
inves tments. 

(3) The concept of "maximization" of some criterion function 
subject to prespecified constraints. For example, maximizing rate of 
return on investment, subject to the budget constraint and the constraint 
that no more than some specified percent of the budget can be spent 

on R&D in any single field. 

Different kinds of quantitative models for project selection are 
formulated on different combinations of these three principles. Models 
based on the first concept are called "Multiple-Criteria Scoring Models." 
Models based primarily on the second concept are called ''Economic 
Models." Models using the third concept are called 
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"Models of Constrained Optimization/* Each of these basic models are 

described below. Combinations are frequent. For exanq>le, maximization along one 

dimension may be only one selection criterion among many; each with 

its own relative weighting. The value of discounted moneys may be another 

criterion, used either in a scoring model or within a maximization 

procedure. Furthermore, different models may be used for different 

purposes; the Scoring Model for selection among candidate projects for 

fundamental research, the Economic Model or Maximization Model for 

advanced engineering or development projects. The complexity of 

each may vary according to the demands of the user and the complexity 

of the candidate projects. Each kind of quantitative model has its 

own underlying procedure, however, and its own set of advantages and 

limitations • 

Scoring Models 

Scoring models provide the most s.lmple framework for comparative 
evaluation among candidate projects. Each project is rated on differ- 
ent criteria, with rating scores associated with different levels or 
ranges along the relevant criteria. For example. Criterion 1 may Ve 
"Probability of Technical Success." If a project is judged to have a 
probability of technical success lying within the range .00 to .01, 
it may receive a score of 1 along that criterion* Similar procedures 
occur for other ranges and other criteria forming the following 
abstract illustration: 

Criterion 1: Probabilitij of Technical Success 

Range 1: (.00 tc .01) Score 1 

Range 2: (.01 to .02) Score 2 

Range 10: (... to .. .) Score 10 

Criterion 2: Expeated Time to Completion 

Range 1: (1 month to 1 year) Score 10 

Range 2: (1 year to 2 years) Score 9 
• • • 

Range 10: (... to ••.) Score 1 
Q Criterion 3: ....... 

ERIC 
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Each criterion may also have a ''relative weighting of importance" 
which allows individual scores along separate criteria to be combined 
intc an overall project score. If the relative weighting of Criterion 
1 is 2, and the relative weighting of Criterion 2 is 1, this indicates 
that the score of a project along Criterion 1 is twice as important 
as the score along Criterion 2. Considering only these two criteria, 
the total project score may be determined according to the overall 
formula: 

Project Score = (Score on Criterion 1) times 2 + Score on Criterion 2 

A different formulation of the overall project score involves multiplying 
component scores rather than adding them. Differences resulting from 
these two procedures are investigated by Moore and Baker in (1) . 

To apply the Scoring Model approach, simply compute an overall 
project score for each candidate project, reorder the list of candi- 
date projects on the basis of decreasing project score, and allocate 
funds for the entire planning horizon to the candidate projects starting 

from the tap of the list. 
Economic Models 

The second model form simply "discounts" the expected streams of 
incomes and expenses for each candidate project and compares them 
according to their computed economic value. In the abstract, the 
model form is: 



Discounted Expected Income 

Project Score - — ^ , z~r"5 

Discounted Expected Expenses 



If this ratio is less than 1, this indicates that considering all the 
flows of income and expenses coming from ia particular candidate project » 
the overall value of the project at the time of initiation is negative. 
Consequently, on economic grounds, no candidate project vhose project 
score is less than 1 should be funded. All candidate projects with 
scores greater than 1 represent investments of positive economic value, 
and the greater the score, the greater the value. Mathematically, the 
formula for determining this ratio is; 
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N 

I (I . (1+r)-^) 

Project Score « K^l 

N 

I (C^ • (l+r)-t) 

where Incoi&e (money gained or costs saved from 

the project in the year t) 

■ Cost of the project in the year t 

r = the discount rate applied to future money 

N = Project lifetime* or number of years into 
the future used in project planning 

t = year (1 N) 

To apply the Economic Model, use the same procedure as before: simply 
compute an overall project score for each candidate project, reorder 
the list of candidate projects on the basis of decreasing project 
score, and allocate funds for tho entire planning hprlzoa to candidate 
projects starting from the top of the list. 

The Economic Model above scores candidate projects on the basis 
of their "present economic value'* considering the flows of all future 
income and •axpenses. This same model may be rewritten in an alternative 
form which scores candidate projects on the basis of their rate of 
return jer dollar invested in the project, or "internal rate of return." 
To calculate project scores using this alternative form of the Economic Model, 
rewrite the above equation so that the ratio is preset to zero, and for 
each candidate project, calculate the resulting rediscount rate r. 

Apply this form of the model in i'\e following way. Suppose that 
the R&D organization which will be funcing candidate projects could 
always reinvest its funds in the bond market at 5% instead of allocating 
Its funds to candidate R&D projects. Consequently, if any candidate 
project has an "internal rate of return" calculated by the second form 
of the Economic Model which xs less than 5%, then this project has 
less economic value than simply putting the same amount of money in 
the bond market. Similarly » if any R&D organization establishes a 
minimum level for rate of return below which it vants to fund no projects, 
this second form of the Economic Model will indicate which candidate 
projects meet that constraint. Projects which do have calculated rates 
^ of return higher than this minimum may then be funded starting from the 

project with the greatest rate of return. 
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Models of Confltrained Optimization 

The third basic model uses the principles of linear programming 
to maximize some specific criterion function subject to multiple 
constraints. Both the criterion and the constraints may be economic 
in nature or may refer to any other measure of relevance or benefit. 
The main difference in concept between the first two models and this 
third one is that in the first two, project scores are independently 
deteritained for each candidate project and the total set of selected 
projects is determined subsequent to the application of the model. 
In this third appro£.ch, the model itself selects the best set of pro-* 
Jects by considering simultaneously all candidate projects. In this 
case, the model £issigns a project score of either 1 or 0 depending on 
whether it includes a project in the set to be funded or in the set to be 
discarded. One example of this model in the abstract is the following: 

N 

maximize the function I *^-f 

i=l 

N 

subject to the constraint Z C. 'x. < b 

i=l ^ ^ 

for each t - l,2,...n 

where V. » Some measure for candidate project i representing 
the degree to which the project accomplishes the 
prespecified criterion. This criterion may be the 
expected economic value, the probability of techni- 
cal success, or any other R&O criterion to be 
maximized. 

x^ = Project score for candidate project i, determined 
by the model to be either: 

1 if the project is selected for funding 
or 

0 if the project is not selected for funding 
C, = Cost of candidate project i in year t 
b^ - Maximum budget for all projects in year t 
N » Number of candidate projects 

n » Number of years in time horizon for project planning 

To apply this third model for project selection, choose the criterion 
to be maximized and rate each candidate project according to the criterion. 
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These ratings are the values V^. Supply the additional required infor- 
mation on candidate project costs and maximum budget levels for the 
future years and use the procedure of linear pro^raumiing to perfotm 
the model calculations. The output of the model will be a list of 
X,, oue for each candidate project. If the value of x. is 1, then 
select candidate project i for funding. If tlie value of x^ is 0, 
then discard the candidate project i from consideration for the current 
planning period. 

Overall Comparison of the Three Model Forms 

Of all three models, only the Economic Model has a focus limited 
usually to considerations of economic value. For financial organizations 
and RU) institutions concerned strictly with economic gain from candi- 
date projects, this model may be very appropriate. For most federal 
agencies sponsoring however, cost and economic gain are only a 

part of a much larger set of criteria used in project selection. In 
these cases, variations of the Scoring Model and Constrained Optimization 
Moxiel are more appropriate. The advantages of the Scoring Model are 
that it is simple and it may not require a computer for the necessary 
calculations. The main advantage of the third approach is that it 
can handle multiple constraints and thus consider different budget 
levels for each year In the planning horizon. To vhow how the Scoring 
Model and Constrained Optimization Model may be focxised on criteria other 
than economic ones, and example of each Is provided below. 

PATTERN: A Scoring Model based on Relevance 

The name PATTERN Is an accronym for Planning Assistance Through 
Technical Eyaliiation of Relevance Numbers. The technique was developed 
by Honeywell, Inc. and is now being tested for application by the U.S. 
Bureau of Mines. The output of the PATTERN Model not a list of 
candidate projects to be funded, but rather a form of roadraap indica- 
ting which technological def Icienv.ies are most Important or cost^ 
effective in terms of meeting various overall objectives and goals of 
the R&D organization. By being able to trace the relations between 
technological def iciences in different areas and organizational goals , 
R&D managers can determine in which project areas it would be most 

ERIC 



advantageous to invests 

Tlie PAXTEEN Model begins with a formulation of iiverall objectives 
provided by a planning team. These overall objectives form the top 
level of what is called a ''relevance tree/' All the lower levels con- 
tain elements which, if accomplished, would contribute to those elements 
above. In the Honeywell version of PATTERN, described by Sigford and 
Parvin (2), the fifth level of the tree outlines various operational 
systems which, if built, would contribute to the accomplishment of 
higher missions and objectives. At the final, eighth level of the 
relevance tree are listed the critical technological deficiencies 
which must be accomplished in order to develop the operational systems 
above. Elements throughout the relevance tree are developed by a 
group of tiechnical experts who discuss whether required advances are 
achievable in the current state-of-the-art, or conceiveable in the near 
future, and how objectives and subgoals would be advanced with improve- 
ments in various technological areas. 

To illustrate the kinds of elements which may be considered, the 
following diagram lists samples of elements from different levels of 
the relevance tree used aa an example in (2). 



Top Level: Overall Objective 
Level 2: 



Level 3 
Level 4 
Level 5 
Level 6 
Level 7 
Level 8 



Operational System 
Functional Subsystem 

Technology Deficiency 



Scientific Preeminence 

Exploration 

Space (or Earth) 

Solar System (or Lunar) 

Unmanned Orbiter (or Solar Probe) 

Navigation (or Power) 

Range and Direction Instruments 

Unreliability of Equipment after 
long shutdown in space 



After the relevance tree has been completed, relating technolo- 
gical deficiencies to objectives and goals, the technical group assigns 
"relevance numbers" to each element according to criteria specified 
for each level of the tree. According to Sigford and Parvin (2), 
example criteria at the levelof operational systems may be Coat 
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Kffectiveness^ Political Implioation, and Scientific Implicationy 
each evaluated In tenos of component characteristics and benefits. 
At the level of technological deficiencies, criteria used may be in 
terms of Feasibility of achieving a solution, Effort needed to solve 
the deficiency, and relative Subaystem Performance Improvement 
achievable per unit of effort spent in advancing a given technology. 
Multiplication of all relevance nuinbers from any one technological 
deficiency up the entire relevance tree produces the total relevance 
number for the deficiency. 

The output of relevance numbers can then be used by R&D managers 
to determine the relative merits of postulated operational systems. 
These merits are in terms of the anticipated contributions of the 
systems toward* overall objectives. Additionally, the relevance 
numbers allow managers to compare tae relative merits of investing 
in different technological deficiencies, 'ijl terms of the importance of a 
deficiency in contributing toward the successful development of the 
postulated systems. 

The PATTERN Technique is a Scoring Model in the sense that numbers 
are assigned to project areas in terms of their ratings along stated 
criteria. The total relevance numbers, or total project area scores, 
are determined by multiplying component rat;inr;s up through the relevance 
tree. The main difference between the PATTERN Model and the abstract 
Scoring Model presented earlier is that criteria in PATTERN are arranged 
hlerarchioally , and the weight given to any criterion is dependent upon 
the ratings of all the elements at higher levels on the tree. 

One feature provided by the PATTERN Technique and usually absent 
in Scoring Models is* a way of measuring cross project impact. Refer- 
ence to the relevance tree in PATTERN will indicate to an R&D manager 
how advances in one area of technological deficiency will effect 
advances required in other areas of deficiency. 

TORQUE; A Constrained Optimization Model based on Utility of Effort 
The name TORQUE is an accronym for Technology Or Research Quantitative 
Utility Evaluation. The technique was developed in the late l960's by 
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an Interservlce team for the Department of Defense, and has been tested 
by the Air Force as a management tool for allocating an R&D budget among 
alternative project areas. A more detailed description of the TORQUE 
Model is presented by Nutt in (3) , 

TORQUE uses the same reasoning found in PATTERN to outline overall 
objectives and determine technological advances required. It goes beyond 
the results of PATTERN, however, in determining the level of funds which 
should be dllocated to each area of technological deficiency in order to 
maximize the overall ''utility** from all the areas funded. 

Like PATTERN, TORQUE begins with a listing of overall objectives. 
The selection of objectives and their rankings in order of importance 
to the R&D organization are provided by planning teams. Other inter- 
disciplinary teams of technical experts develop alternative approaches 
to realizing the objectives, and identify the technological advances 
required. In a procedure analogous to providing **relevance numbers,** 
TORQUE planners assign ratings on the criticality of a technology area 
to the approach or system which it supports. 

The next steps extend the scope of TORQUE into the area of resource 
allocation. The first task is for the teams technical experts to 
sort technological areas into related groups and divide the groups into 
sequential levels of difficulty. Next, the technology teams determine 
the resources required to achieve the various technologies identified. 
The final step is to determine the best allocation of funds in support 
» of different Levels of Difficulty (LOD) for each technology area. 

To determine the best use of funds, the TORQUE Model calculates, 
for each Level of Difficulty for a technology area, a utility score, 
defined as: 

N 

U = E C "W.-Cf-t. 
1=1 ^ ^ 

where U = **Utility ** of achieving a particulan Level of Difficulty 
for a particular technology area 

N - Number of systems or approaches supported by the Level 
of Difficulty 

C * **Criticality** of the Level of Difficulty to the i^h 
^ system or approach supported . (Criticality ratings 

range from 0 as "no contribution" to 1.0 as **absolutely 
Q essential .**) 
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provide an Indication of the variety of management systems possible in 
conducting large scale efforts in programcoatic R&D. 
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will be running the program after Ir starts. At NCI this man Is called 
a Science Maaager. Other men on the planning team should be specialists 
In relevant technical fields. Their fields may or may not overlap, and 
the men may be either intramural or extramural scientists. The last 
person on the planning team should be from the systems analysis staff. 
His role Is one of synthesizer and as such he must he experienced In the 
planning technique. 

According to Mr. Carrese, some Important informal rules In bring- 
ing together this team are: Never have more than seven on the team. Take 
notes at the first few sessions, but do not produce a transcript. 

The first job is to set down an operational objective which, if 
achieved, will clearly end the program. An example of an unoperational 
objective is: Cure leukemia. An example of an operational objective 
Is: Develop a vaccine for acute early childhood leukemia. Typically, 
formulating the program objective takes from one to five days, but 
sometimes takes much more time. One common impulse in beginning 
program planning is to survey the literature. At all costs, suggests 
Mr. Carrese, this must be avoided. First layout the overall program 
objectives. Then do the literature search. 

For the next three to five weeks; lay out the intermediate objec- 
tives, decision pionts , and steps. (These are defined below.) The 
attitude In these meetings must be a willingness to forget about re- 
source coi!istralnts. During this tlme» the systems analysis staff 
should help the team build the program plan (called a "Convergence 
Chart") and keep them on a productive track. After a plan Is prepared, 
invite a lot of experts In the field to criticize and help modify it. 
Encourage constructive revision. The end product is a program plan to 
be used by the program manager and his segment chairman. 

THEORY OF THE CONVERGENCE TECHNIQUE 

The logical process by which a convergence chart Is madci is back- 
ward directed, from end goal to present state. To reach the end goal, 
several Intermediate goals will have to be reached. Each Intermediate 
goal is the title of a program phase . Together, the intermediate goals 
Indicate the "mlnimunr critical work needed to achieve the program objective 
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It takes a great deal of discussion and analysis in the planning meet- 
ings to reach this point. 



Phase 1^ 



Phase 2 



-> 



Phase 3 



The next logical question is: "How vill we know when an intermed- 
iate goal has been reached?" Determining the accomplishment of an 
intermediate goal is defined as a program decision and has associated 
with it a set of minimum acceptable criteria that will be used in 
making that decision. 



Phase 
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The criteria indicate what data will be needed in order to make a 
particular program decision. In turn, data requirements and inter- 
mediate goals provide clues to the research steps that must be under- 
taken. 



iPhase 



Step 1 



Step 2 



Step 3 



Knowing these steps, it is usually possible to formulate the project 
areas In which work is required for each step. 



Phase 



Step 1 

j Project Area 1 
Project Area 2 




ERIC 



The logical sequence of these boxes is called The Linear Array. 
The primary reason for doing research along a linear array is to obtain 
data required to meet criteria for making a program decision. Together, 
these program decisions direct research activity toward the predefined 
program goals. Using this framework of planning, "managing" simply 
means ensuring the acquisition of data needed to make program decisions* 
Accordingly, managers need "data progress" reports rather than pro-* 
ject status or change reports, since neither of the latter relate 
directly to program decisions. 

A second planning array (presented below the linear array on a 
convergence chart) is of Concurrent Reeeardh. This is a set of pro- 
gram-related research areas, not required in the program logic, but 
which may yield important insights. In allocating the program budget, 
projects in these areas have second priority. 

A third array (below the second) is the Supplementary Array^ 
which includes "blue sky" project areas that may have big impacts, but 
with low probability. Many of these projects in these areas are funded 
as grants in the normal NIH manner. 

Next to each project area described for a program step in the linear 
array is an open circle i£ work is required to be done, and a solid 
dot if substantial work is underway or completed. This provides an 
efficient way of exposing the program manager to his current and fu- 
ture needs. Work in open circle project areas must eventually be 
conducted because all project areas on the linear array have been iden- 
tified as among the minimal acceptable, set of project activity to be com- 
pleted, according to the team of experts who have developed the linear 
array . 

7:he final planning step is to "cluster" related project areas into 
"segments" for the purpose of efficient prograi? management. Seg- 
ments may be constructed according to methodology, field, equipment 
requirements, or foc\-s. Many will cut across organizational branches 
of the division. 

An abstract of the program plan ("Convergence Chart") prepared 
for the Cancer Chemotherapy Program (CCP) is illustrated in the 
following figure. A more detailed view of part of this Chart is 



presented In the subsequent figure. This second figure provides an 
example of criteria specified for a particular program decision 
(surrounded by an evenly dotted line) as well as the flow of program 
activity determined by more broad program decisions by the Program 
Director. 



\ 
\ 
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APPENDIX B 

ABSTHACT DESCRIPTION OF THE AIR FORCE 
PROCESS OF PROJECT INITIATION AND PLANNING 

INTRODUCTION 

The following eight diagrams illustrate the various steps and flow 
of documentation usually involved in the initiation and planning of any 
new Air Force major weapon development project, under both the old 
regulations and the new. These steps and flow are superimposed on a 
partial chart of the Air Force organizational structui.e which highlights 
the relevant portions participating in the processes. 

Materials from which this description is drawn include Air Force 
Regulations AFR-57-1, the p^ist AFR-j?5 series now superceded by AFR 
800-2, Department of Defense Directives 3200 and 5010, and interviews 
with Major Thomas Tiemey, RAND Air Force liason officer, and with the 
System Project Director and members of his staff for the SRAM Missile 
Development Project. 

Not all Air Force projects follow these procedural i^teps. Depend- 
ing on the importance of the pi^oject idea and on its relevance to other 
already existing projects, some of the steps may be eliminated or signif- 
icantly shortened. The entire description, however, provides a general 
picture of the typical initiation and planning steps required in project 
management under the system used by the Air Force over most of the last 
decade, and emphasizes the use of hierarchical tiers of multiple review. 
Where appropriate, major revisions consistent with the new regulations 
are Indicated. 

DESCRIPTION OF THE TYPICAL STEPS 

STEP 1: PREPARATION OF ROC 

A perceived military need or deficiency Is stat^i^d on a "Letter of 
Transmittal" from any general officer within the Air Force operating 
divisions, such as the Strategic Air Command or the Tactical Air Command^ 
to the officer's Commander. This letter is forwarded to the ComDiander's 
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Figure 13: 

Step 1 : ROC Prepared 
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Dlrector of Requirements who prepares a statement of the project idea 
in the form of a Required Operational Capability (ROC). See Figure 13. 

STEP 2! OOORDINAnON OF ROC 

The Director of Requirements located within the operating division 
of ROC origin sends the ROC to the Directors of Requirement of all the 
other divisions in order to prevent duplication of efforts and to 
foster ^oint benefits among all proposed projects. Se^^ Figure l4. 

STEP 3: ROC APPROVAL 

Following return of the comments from all the Directors of Require- 
ment, the ROC is sent to the Commander of the originating division for 
approval. See Figure 15. 

STEP 4: PREPARATION OF POSITION PAPER UNDER THE OLD SYSTEM 

If the Commander chose to approve the ROC as a project idea 
worthy of subsequent action, the ROC was sent to Air Force Headquarters 
and was received by the division of Operational Requirements and Develop- 
ment Plans (AFRDQ) located under the Deputy Chief of Staff of R&D. 
Here it was assigned to an Action Officer responsible for review manage- 
ment. 

The Action Officer first sent the ROC in its current form to all 
the Deputy Chiefs of Staff and to the appropriate formally constituted 
review panel maintained continuously within AFRDQ. One of these panels 
existed for each of the Strategic, Tactical, and Airlift program areas 
of the Air Force. Preliminary comments from these personnel were then 
returned to the Action Officer who synthesized their ideas and evalua- 
tions into a Position Paper. Both the ROC and its associated Position 
Paper were then redistributed to the same reviewers for statements of 
concurrence or non-concurrence with the Position Paper as written. See 
Figure 15. (Details of the new program management procedures are not 
yet defined.) 

STEP 5: - FIRST MAJOR HEADQUARTER REVIEW UNDER THE OLD SYSTEM 

The Position Paper along with all the responses to it by the 
Deputy Chiefs of Staff were forwarded to the same area-focused panel 

ERIC 
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Figure 14: 
Step 2: Coordination of ROC 
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Flgure 15: 
Step 3: ROC Approval 
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Figure ?^ : 

Step 4: Preparation of Position Paper under the Old Systero 
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within AFROQ for the first major headquarrsr revlevr. See Figure 17. 

STEP 6: PREPARATION OF VARIOUS RADs UNDER THE OLD SYSTEM 

Panel approval of the ROC was followed by a succession of Required 
Action Directives (RADs) which called for supplementary studies involving 
either mission analyses, technical feasibility of the project idea, or 
analyses of current technology for possible application to the project 
idea. The majority of these studies were performed extramurally under 
the management of Air Force Systems Command. At this point, overall 
management of the project idea review was given to a Program Element 
Monitor who monitored the execution of all RADs and kept the Panel 
informed of study progress. See Figure 18. (Under the new program 
managerment regulations, RADs and other headquarter directive documents 
are replaced by the single Program Management Directive.) 

ST EP 7: SECOND AND THIRD HEADQUARTER REVIEWS UNDER THE OLD SYSTEM 
Prevlotisly, RADs issued by ARFDQ have kept some project ideas 
within the study stage for a nunber of years, l^ollowing execution of 
all RADs ordered by the Panel, the original ROC plus all its supporting 
studies received two more headquarter reviews, one by the Air Staff 
Board, consisting of all the department heads under the Deputy Chiefs 
of Staff, and one by the Air Force Council, consisting of the Deputy 
Chiefs themselves. 

Approval by both groups was required for the project to proceed. 
If it were granted, a final RAD was issued by the Panel to Systems 
Command for the preparation of the project plan. This plan, called thv^ 
Concept Formulation Package/Technical Development Plan, was prepared 
esi^trr'^'nurally under the management ot Systems Command. See Figure V). 
(In the new system, the same two groups hold project reviews, but the 
scope of the project plan is substantially reduced and responsibility 
for plan preparation is given to the Program Manager appointed by 
the implementing conmand rather than by Systems Command.) 



STEP 8: PREPARATION OF THE PROJECT PLAN 

To help prepare the former project plan (CFP/TDP) , Systems Command 
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Figure 17: 

Step 5: First Major Headquarter Review under the Old System 
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Figure 

Step 6: Preparation and Execution of RADs 
for Supplementary Studies 
u::rder the Old System 
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Fi^ure 19: 

Second and Third Headquarter Reviews 
under the Old System 
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appolnted a System Project Director and a cadre of five or six personnel 
to form the core of a System Project Office (SPO) . This SPO prepared 
the REPs for the extramural development of the project plan ( and if 
the project was funded by Congress, also managed the overall extramural 
system development) ; Representatives from AFRDQ and Logistics Command 
were included in the SPO team to help prepare the RFPs for the project 
plan. See Figure 20* (Preparation under the new regulation is the 
responsibility of the Program Manager only.) 

In the former system, following preparation of the Concept Formu- 
lation Package/Technical Development Plan were additional reviews by the 
Air Staff, Air Force Council, and the Office of the Secratary of De- 
fense. Approval by each, and authorization for funding by Congress 
did not* however, end the overall planning process. Prior to develop- 
ment and distribution of RFPs, evaluation of proposals, and awarding 
of con tz acts for actual system development was the production of volumes 
of detailed specifications of design and management procedures which 
made up the actual contract content. 

Contract terminology was omitted in a more operational planning 
document of General Specifications for Performance and Design Requirements 
which was used by the system developer as a manual of minimal tasks and 
capabilities which must be achieved to fulfill the contract agreements. 
These General Specifications included both minimum operational require- 
ments and minimum proceudras of testing and analysis to determine 
whether the requirements were achieved. 

From the viewpoint of the contractor, all operational requirements 
and procedures of quality assurance stated in the General Specifications 
Plan were contractual commitments. In no way was he free to change that 
plan. Even the System Project Director responsible for the overall 
management of the project was unable to change the plan. Any changes in 
either the minimum levels of operational performances required, in the 
overall cost of the project, or in the time stipulated for development 
or production phases had to be approved by most of the participant 
groups involved in the original project planning process. Consequently, 
approvals for changes of these kinds Involved lengthy procedures of 
additional multiple reviews and were sought only if It became clear that 
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mininmin requireinents would not be achieved within the time and cost 
constraints* Minor specifications of design, however, which did not 
adversely affect either time, cost, or performance levels could be 
enacted by the System Project Director as a result of communication 
and joint decisionmaking between his staff of engineers within the 
System Project Office and the engineers of the development contractor. 

Under the new regulation of July, 1971, much of these procedures 
are retained. Though greater responsibility for project development 
is given to the Program Manager, he nust still gain higher approval 
for any changes effecting overall cost, time, or performance beyond 
the stated requirements. 



